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JThis  report  describes  research  and  development  centered  on  evaluation  of 
information  needs  in  advanced  aircraft  operations.  The  selection  of 
information  for  display  is  a  recurrent,  subjective  decision  involving  many 
factors--aircraft  state,  environmental  conditions,  operator  capabilities, 
and  acquisition  costs,  among  others.  An  adaptive  computer  model  has  been 
developed  which  incorporates  these  factors  into  a  multi-attribute  decision 
model.  The  program  is  designed  to  capture  the  operator's  information  seeking 
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r'1  policy  using  a  training  algorithm  based  on  pattern  recognition -techniques. 

*  The  individual  policy  is  then  used  for  information  system  evaluation  and 
for  automated  information  management. 

Experimental  tests  of  the  adaptive  modeling  and  information  management 
approaches  were  made  using  a  task  simulation  resembling  multiple  intercept 
operations  in  advanced  aircraft.  Individual  subjects  (8  in  study) 
navigated  a  simulated  aircraft  through  a  hazardous,  changing  environment.^ 

In  doing  so,  the  operators  selected  from  five  Information  sources  of 
varying  content,  cost,  delay  and  detectability.  The  information  was  then 
used  to  take  one  of  a  number  of  control  actions.  Resources  were  constrained 
in  the  task,  so  that  a  session  terminated  when  the  available  costs  were 
exhausted.  Two  forms  of  automated  information  management  were  used,  one 
based  on  a  multi-attribute  utility  (MAU)  model  and  one  based  on  a  cost- 
benefit  model.  Both  models  were  trained  adaptively  from  the  manual 
Information  selection  behavior  of  each  individual.  No  significant  differences 
in  performance  score  between  the  two  forms  of  automated  information 
selection  were  noted,  although  the  MAU  model  was  significantly  more  cost 
conserving  and  accurate  in  information  value  analysis.  Operator  responses 
-to  differing  levels  of  .infcrnption  complexity  and  speed  stress  were  also 
determ1nea.'^?The  report^eKw^Wtto  an  example  application  of  the  information 
management  program  to  F- 14  threat  intercept  operations  and  w+th“a  set  of 
system  application  guidelines. 


UNCLASSIFIED 


SECURITY  CL  A  SSI  PiC  AT  ION  Of  rMlS  D»tm  Fnfrmd) 


1 


l 

l 

E 

f. 


Fi/isUr 


Technical  Report  PATR-1037-79-6 
June,  1979 


/^Wt  j>.  -  ?c  c  -c.  // 


ADAPTIVE  ESTIMATION  OF  INFORMATION  VALUES 
IN  CONTINUOUS  DECISION  MAKING  AND  CONTROL 
OF  ADVANCED  AIRCRAFT 


RANDALL  STEEB 
YEE-YEEN  CHU 
CYNTHIA  CLARK 
YORAM  ALPEROVITCH 
AMOS  FREEDY 


I. 

i. 

. 

i: 

c 

i 

i 

i 


Prepared  For: 

AIR  FORCE  OFFICE  OF  SCIENTIFIC  RESEARCH 

Bolling  Air  Force  Base 
Washington,  D  C.  20333 


PERCEPTRONICS 


AIK  F(  RCK  <  FF1  7E  OF  SCI . 

NOTICE  01 . ITTAI 

Tt: ; 1 

nppc,  •  ..... 

Distnfcui  i . .  •••*.. 

A .  D.  *ua  - 

Tochulc.il  luforavit  -cu  Ol'l’icur 


■I  l::sc) 

i  ^3 


b). 


•271  V ARIEL  AVENUE  •  WOODLAND  HILLS  •  CALIFORNIA  91367  •  PHONE  (213)  M4-7470 


TABLE  OF  CONTENTS 


PAGE 

1.  INTRODUCTION  1-1 

1.1  Summary  1-1 

1.2  The  Problem  1-2 

1.3  Technical  Approach  1-3 

1.4  Current  Objectives  1-4 

1.5  Applications  1-5 

1.6  Report  Organization  1-6 

2.  INFORMATION  VALUE  MODELING  IN  ADVANCED  AIRCRAFT  OPERATIONS  2-1 

2.1  Overview  2-1 

2.2  Information  System  Characteristics  2-2 

2.3  Structure  of  the  Information  Seeking  Models  2-3 

2.3.1  The  Multi -Attribute  Models  2-3 

2.3.2  Other  Methodologies  2-5 

2.3.3  Information  Management  Functions  2-7 

2.3.4  Attribute  Development  2-14 

2.3.5  Consequence  Level  Determination  2-17 

2.3.6  Attribute  Weight  Estimation  2-18 

2.3.7  Probability  Estimation  2-22 

2.3.8  Level  of  Detail  2-23 

2.4  Aiding  in  Advanced  Aircraft  Operations  2-24 

2.4.1  Forms  of  Aiding  2-24 

2.4.2  System  Evaluation  2-24 

2.4.3  Automated  Management  of  Information  2-25 

2.5  Experimental  Validation  2-26 

3.  EXPANDED  SYSTEM  DEVELOPMENT  3-1 

3.1  Limited  Resources  and  the  Use  of  Cost-Benefit 

Decision  Criteria  3-1 

3.2  Continued  Sampling  of  Information  3-3 

3.3  Process  Model  of  Information  Seeki 

3.4  Operator  Loading 


-  >  / 


TABLE  OF  CONTENTS  (CONTINUED) 


PAGE 

4.  EXPERIMENTAL  STUOY  4-1 


4.1 

Overview 

4-1 

4.2 

Hypotheses 

4-1 

4.3 

Task  Simulation 

4-2 

4.3.1  Displays  and  Controls 

4-2 

4.3.2  Information  and  Action  Options 

4-4 

4.3.3  Situational  Conditions 

4-7 

4.4 

Decision  Model 

4-9 

4,4.1  General 

4-9 

4.4.2  Decision  Attributes 

4-11 

4.4.3  Model  Form 

4-12 

4.4.4  Performance  Measures 

4-14 

4.4.5  Subjects  and  Procedure 

4-15 

EXPERIMENTAL  RESULTS  AND  DISCUSSION 

5-1 

5.1 

General  Observations 

5-1 

5.2 

System  Performance 

5-1 

5.3 

Performance  Components 

5-3 

5.4 

Aiding  and  Task  Loading 

5-3 

5.5 

Information  Value 

5-8 

5.6 

Subjective  Responses 

5-8 

6.  CONCLUSIONS  AND  RECOMMENDATIONS  6-1 

7.  EXAMPLE  APPLICATION:  F-14  THREAT  INTERCEPT  OPERATIONS  7-1 


7.1  Background 

7.2  Problem  Area 

7.3  LAR  Aiding  System  Concept 

7.4  Decision  Aid  Configuration 


7-1 

7-1 

7-2 

7-5 


7.4.1  Problem  Identification  7-5 

7.4.2  Problem  Display  7-5 

7.4.3  Threat  Evaluation  Computation  7-6 

7.4.4  Maneuver  Assessment  7-6 

7.4.5  LAR  Entry  Decision  Criteria  7-7 


i  i 


TABLE  OF  CONTENTS  (CONTINUED) 


PAGE 

7.5  Information  Requirements  of  the  Aiding  System  7-10 

7.6  Feedback  Options  ’  7. 12 

8.  REFERENCES  8.1 

APPENDIX  A:  SYSTEM  APPLICATION  GUIDELINES  A-l 


111 


1.  INTRODUCTION 


1.1  Summary 

This  report  covers  the  third  year  of  a  three-year  program  of  research  and 
development  directed  toward  evaluation  of  information  needs  in  decision 
making  and  control  of  advanced  aircraft.  Its  purpose  is  to  establish  new 
techniques  for  the  selection  of  essential  information  in  operational 
systems.  These  techniques  center  on  the  adaptive  modeling  of  individual 
decision  behavior  to  derive  the  immediate  value  of  information.  The 
program  is  designed  to  provide  the  framework  for  development  of  operational 
aids  for  the  real  time  management  of  communications.  Specific  objectives 
of  the  three-year  program  include  the  following: 

(1)  Establish  a  repertoire  of  techniques  to  model  the  information 
needed  for  operation  of  tactical  airborne  systems. 

(2)  Provide  model -based  computer  programs  capable  of  ascertaining 
the  potential  usefulness  of  alternative  information  sources, 
transmission  systems,  and  information  displays. 

(3)  Develop  and  experimentally  validate  a  supervisory  program 
for  management  of  communications  in  simulated  tactical  air¬ 
borne  systems. 

(4)  Produce  guidelines  for  field  application  of  the  information 
analysis  and  management  programs  in  a  variety  of  complex 
systems. 

The  initial  year’s  work  established  techniques  for  modeling  the  information 
seeking  decisions  necessary  for  continuous  decision  making  and  control  of 
remotely  piloted  vehicles.  The  work  also  experimentally  demonstrated  the 
usefulness  of  adaptive  techniques  for  prediction  and  analysis  of  informa¬ 
tion  seeking  behavior. 
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The  second  year  of  the  program  focused  on  the  application  of  adaptive 
information  model  in  simulated  advanced  aircraft  operations.  The  program 
demonstrated  the  usefulness  of  the  adaptive  decision  model  for  automating 
the  presentation  of  information  and  for  evaluating  the  effectiveness  of 
information  display  configurations.  The  work  reported  here,  the  third 
year  of  the  program,  expands  the  domain  of  the  information  value  model 
to  include  situations  of  resource  limitations,  continued  sampling  of 
information,  and  task  loading.  The  program  also  involves  the  incorporation 
of  the  multi-attribute  decision  model  into  a  larger  process-oriented  model 
of  information  management.  The  work  ends  with  an  example  application  to  an 
operational  system,  the  F-14  AWG-9  weapon  system,  and  with  a  set  of 
generalized  guidelines  for  application. 

1.2  The  Problem 

Command  and  control  of  modern  military  systems  involves  selection  among 
increasing  large  amounts  of  information.  Computerized  systems  typically 
make  available  copious  amounts  of  information  concerning  remaining 
resources,  environmental  state,  potential  computer  aiding,  and  predicted 
circumstances  and  actions.  The  costs  of  coimunications  and  the  limited 
processing  capabilities  of  the  human  operator  make  it  necessary  to  optimize 
the  information  selected,  transmitted  and  presented. 

The  central  problem  in  performing  an  analysis  of  information  needs  is  the 
structuring  of  the  decisions.  Choices  must  be  modeled  regarding  variables 
such  as  the  mix  of  information  sensing,  processing,  encoding,  transmitting, 
and  display.  Throughout  this  process,  a  balance  must  be  maintained  between 
maximizing  operator  awareness  of  system  operation  and  minimizing  communi- 
cations  costs  and  operator  load. 


Some  initial  efforts  have  been  made  toward  analyzing  and  automating  the 
communications  management  functions.  Information  and  control  allocation 
techniques  have  been  proposed  using  criteria  based  on  queing  models 
(Rouse,  1975;  Engstrom  and  Rouse,  1976;  Steeb  and  Chu,  1978),  optimal 
control  models  (Sheridan,  1976;  Rouse  and  Gopher,  1977),  and  multi¬ 
attribute  decision  models  (McKendry,  Enderwick  and  Harrison,  1971;  Steeb 
and  Freedy,  1976).  This  program  represents  an  effort  to  develop,  inte¬ 
grate  and  implement  the  more  promising  of  these  techniques. 

1.3  Technical  Approach 

In  brief,  the  command  and  control  task  can  be  represented  as  a  multi¬ 
stage  decision  task  of  information  acquisition  and  action  selection.  The 
effectiveness  of  the  action  decisions  are  dependent  on  the  appropriateness 
and  timeliness  of  the  information  acquired.  The  choices  of  what  informa¬ 
tion  to  transmit  thus  must  reflect  the  task  circumstances,  the  operator's 
and  automatic  system's  capabilities,  and  the  communication  channel 
characteristics.  These  decisions  can  be  expressed  analytically  using  a 
multi-dimensional  set  of  utilities  tied  to  the  potential  consequences  of 
the  actions.  The  weights  of  the  various  dimensions  reflect  the  importances 
placed  by  the  individual  decision  maker  on  each  factor. 

A  program  was  developed  along  these  lines,  containing  both  objective  pro¬ 
bability  estimation  algorithms  and  subjective  behavioral  models.  The  pro¬ 
bability  estimation  algorithms  calculate  the  likelihood  of  occurrence  of 
consequences.  Such  algorithms  depend  on  frequency  determinations  and 
optimal  Bayesian  calculations.  The  subjective  models  use  these  determina¬ 
tions  as  inputs  and  are  adaptive  multi-attribute  and  cost-benefit  forms 
tied  to  prediction  of  behavior.  The  models  demonstrate  potential  for  a 
number  of  types  of  aiding.  Among  the  uses  planned  are  evaluation  and 
specification  of  conmuni cation  configurations,  individualized  training  in 
information  acquisition,  and  automated  communications  management. 


Testing  of  the  models  was  accomplished  by  determining  the  aiding  and 
analysis  capabilities  provided  in  an  advanced  aircraft  task  simulation. 

The  task  simulation  incorporates  many  of  the  functions  involved  in 
multiple  intercept  operations  and  has  extensive  performance  analysis 
capabil ities. 

1.4  Current  Objectives 

The  focus  of  this  phase  of  the  research  and  development  program  is  to 
extend  the  application  of  the  adaptive  decision  model  to  include  informa¬ 
tion  evaluation  and  management  functions  in  an  advanced  aircraft  context. 

This  builds  on  the  findings  of  the  first  and  second  year's  work  in  which 
the  adaptive  program  was  found  to  be  a  useful  model  for  analyzing  and 
predicting  information  seeking  behavior,  for  aiding  the  information  choices, 
and  for  evaluating  the  usefulness  of  alternative  information  system  confi¬ 
gurations.  The  specific  objectives  of  this  third-year  program  include: 

(1)  Expand  the  domain  of  the  information  value  model  to  include 
situations  of  resource  allocation  and  continued  sampling  of 
information. 

(2)  Incorporate  the  information  value  model  into  a  process  model 
of  information  management:  analytically  relate  information 
sources  to  attributes,  attributes  to  decisions,  and  decisions 
to  users. 

(3)  Expand  the  task  simulation  to  include  limited  resources  and 
operator  loading.  Ascertain  unloading  provided  by  automated 
information  management  based  on  multi-attribute  utility  (MAU) 
and  cost-benefit  models. 

(4)  Produce  guidelines  for  decision  model  choice,  model  structuring, 
and  estimation  techniques;  and  describe  methods  of  application 
to  information  system  evaluation  and  management. 
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1.5  Appl i cations 

Model-based  aiding  for  information  value  selection  appears  to  be  most 
applicable  to  decision  tasks  which  feature  some  or  all  of  the  following 
operational  characteristics: 

(1)  High  Information  Load.  The  operator  is  in  a  time-stressed 
decision  task.  For  each  decision  he  can  process  only  a 
portion  of  the  available  data  set  and  must  choose  an  action 
within  a  short  time. 

(2)  Costly  Information  Transmission.  The  transmission  of  data 
to  the  operator  is  subject  to  cost,  risk  of  detection,  or 
limited  transmission  capabilities.  Immediately  valuable 
information  must  be  selected. 

(3)  Significant  Judgmental  Factors.  The  decision  maker  must 
consider  the  credibility  and  content  of  the  evidence  along 
with  the  probabilities  and  utilities  of  the  consequences 
associated  with  each  ensuing  action. 

Among  the  examples  of  actual  military  and  civilian  decision-making  situa^ 
tions  which  require  such  tasks  are: 

(1)  Advanced  aircraft  decision  making. and  control. 

(2)  Remotely  piloted  vehicle  guidance. 

(3)  Satellite  intelligence  transmission. 

(4)  Supervision  of  air,  ground,  or  sea  support  operations. 

(5)  Crime  information  management. 

(6)  Coordination  of  remote  sensing  of  the  environment. 
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1 .6  Report  Organization 


The  organization  of  this  report  is  as  follows:  Chapter  2  reviews  selected 
background  concepts,  traces  the  development  of  the  adaptive  models,  and 
describes  previous  experimental  studies  performed  under  this  program. 
Chapter  3  discusses  the  expansion  of  information  value  modeling  to 
situations  of  limited  resources,  varying  operator  loading  and  continued 
sampling  of  information.  Chapter  4  describes  the  experimental  plans  and 
procedures  for  testing  the  usefulness  of  the  adaptive  MAU  and  cost-benefit 
models.  Chapter  5  summarizes  the  results  of  the  experimental  study. 
Chapter  6  presents  a  discussion  of  the  research  findings  and  opportunities 
for  application.  Chapter  7  discusses  use  of  the  model  in  an  operational 
setting:  flight  officer  aiding  in  the  F- 14  aircraft.  The  report  ends 
with  system  application  guidelines  sunmarizing  the  findings  of  the  three 
years  of  work. 
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2.  INFORMATION  VALUE  MODELING  IN  ADVANCED  AIRCRAFT  OPERATIONS 


2.1  Overview 


The  selection,  acquisition  and  processing  of  information  are  activities 
that  are  involved  in  virtually  every  aspect  of  tactical  airborne  operations. 
The  operator  must  continuously  maintain  an  awareness  of  the  aircraft  state, 
the  environment,  the  capacity  and  quality  of  the  communication  channels, 
and  the  progress  toward  objectives.  The  operator  must,  under  considerable 
time  stress,  weigh  the  probable  usefulness  and  costs  of  a  variety  of 
competing  forms  of  information--mission  status,  track  data,  environmental 
information,  aerodynamic  functioning,  etc.  These  moment-to-moment  judgments 
must  often  be  based  on  subjective  factors,  since  the  decision  is  normally 
too  complex  and  dynamic  to  be  analytically  tractable. 

Adaptive  decision  models  based  on  multi-attribute  utility  analysis  have 
been  employed  in  earlier  studies  (Steeb,  Chen  and  Freedy,  1977;  Steeb, 

Davis,  Alperovitch  and  Freedy,  1978)  to  model  the  information  seeking 
decisions.  These  techniques  were  found  to  hold  a  great  deal  of  promise  for 
aiding  in  advanced  aircraft  operations.  Tactical  airborne  systems  typically 
require  consideration  of  voluminous  amounts  of  relevant  and  irrelevant  infor¬ 
mation.  At  the  same  time,  severe  time  stresses  and  interfering  tasks  can 
degrade  the  information  integration  processes.  To  reduce  the  operator  load, 
combat  aircraft  typically  have  sophisticated  autopilot  and  weapon  control 
systems.  These  systems  have  been  shown  to  be  extendable  to  include  functions 
of  information  evaluation  and  management,  since  they  take  into  account  the 
key  factors  of  environmental  state,  vehicle  state,  and  task  objectives.  The 
following  sections  describe  the  initial  definition,  structuring,  and  imple¬ 
mentation  of  adaptive  decision  models  in  the  context  of  tactical  airborne 
operations. 
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Information  System  Characteristics 


The  typical  advanced  aircraft  mission  can  be  defined  by  a  series  of  mission 
phases,  much  like  the  stages  of  the  RPV  supervision  task  analyzed  in  the 
initial  year  (Steeb,  Chen  and  Freedy,  1977).  The  phases  can  be  characterized 
by  the  danger  or  frequency  of  threats,  the  time  available  for  decision 
making,  and  the  options  and  characteristics  of  information  concerning  the 
aircraft  and  the  environment. 

The  information  available  at  a  given  time  is  dependent  on  the  environmental 
situation,  the  sensor  characteristics,  the  data  base  content,  and  the  dis¬ 
play  capabilities.  The  information  itself  may  consist  of  data  regarding 
weather  conditions,  aerodynamic  status,  target  track,  ECM,  and  mission 
status. 

The  costs  of  acquiring  information  result  from  the  sensor  characteristics, 
the  direct  and  indirect  costs  of  sensor  deployment,  information  processing 
and  display,  and  the  amount  of  attention  the  operator  can  contribute.  The 
direct  costs  of  information  acquisition  include  such  factors  as  energy 
expenditures  and  equipment  expenses.  Indirect  costs  include  increased 
possibilities  of  detection  and  countermeasures.  The  available  operator 
attention,  finally,  is  defined  by  the  task  demands  and  the  individual  capa¬ 
bilities  of  the  operators. 

The  costs  and  payoffs  associated  with  the  various  possible  outcomes  vary 
with  mission  phase.  The  consequences  are  defined  not  only  in  terms  of 
attrition  of  equipment  and  attainment  of  objectives,  but  also  as  a  function 
of  organizational  policy  and  procedures.  The  relative  importance  of  fuel 
expenditures,  vehicle  survival,  countermeasures,  etc.,  change  as  the  mission 
objective  is  approached,  attained,  or  past.  The  relative  importance  of 
these  factors  must  be  assigned  by  the  human  operator  or  by  the  conmand 
group. 
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Available  time  for  decision  making  varies  throughout  the  mission  as  a 
direct  function  of  the  varying  vehicle  speed,  altitude,  and  surrounding 
weather  conditions.  Altitude,  cloud  cover  and  ECM  determine  the  distance 
that  obstacles,  navigation  points,  or  targets  can  be  observed.  The  speed 
then  determines  the  available  time.  Decision  time  can  be  expected  to 
influence  the  amount  of  information  that  can  be  processed  and  the  proba¬ 
bility  distribution  of  the  possible  consequences. 

The  pilot  as  the  airborne  system  manager  faces  a  variety  of  information 
sources  and  displays--such  as  Master  Monitor  Display,  Integrated  Multi- 
Function  Display,  etc.  The  pilot  has  the  responsibility  to  monitor  the 
aircraft  subsystems  as  well  as  supervise  the  autopilot  and  to  detect 
possible  hardware  failures  and  potential  hazards.  He  must  constantly 
respond  to  action-evoking  events  such  as:  communication  of  information, 
change  of  aircraft  configuration,  and  reduction  of  4-D  guidance  errors. 
Finally,  the  pilot  is  required  to  respond  to  unexpected  events  such  as 
identification  and  avoidance  of  threats,  change  of  flight  plan,  establish¬ 
ment  of  the  backup  mode,  and  declaration  of  emergencies,  etc.  The  pilot 
is  in  a  multi-information  selection  situation. 

In  sum,  the  selection  of  information  and  control  to  allocate  to  the 
supervisory  human  operator  is  a  complex  and  dynamic  decision.  The  decision 
maker  must  continually  weight  the  probable  usefulness  of  the  information 
against  the  costs  of  acquiring  it.  Since  this  decision  is  especially  diffi 
cult,  the  costs  and  benefits  are  both  multi-dimensional  and  probabilistic. 


Structure  of  the  Information  Seeking  Models 


2.3.1  The  Mul ti -Attribute  Models.  Multi-attribute  utility  (MAU)  models, 
pioneered  by  Raiffa  and  his  colleagues  (Raiffa,  1969;  Keeney  and  Raiffa, 
1975)  and  by  V.  Winterfeldt  (1975)  provide  a  framework  for  aiding  the 
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information  management  process.  These  utility  models  tie  the  information 
decisions  directly  to  the  ensuing  action  decisions.  The  value  of  obtaining 
information  is  determined  by  calculating  its  impact  on  the  expected  utility 
of  the  subsequent  action  decision.  The  information  is  assumed  to  change 
the  probability  distributions  of  the  consequence  sets  and,  in  turn,  to 
revise  the  expected  values  of  the  alternative  actions.  Nevertheless,  the 
form  of  the  model  is  again  a  linear  additive  rule.  The  utility  of  an 
action  is  considered  to  be  an  aggregate  of  many  possible  outcomes,  each 
expressed  along  a  set  of  attributes: 

EU  (ak)  =  l  P(zh)  l  Ui  (ak,  zh)  (2-1) 

states  attributes 
h  i 

Where  EU  (ak)  is  the  expected  utility  of  action  k,  P(zh)  is  the  probability 
of  state  z^  occurring,  and  (ak,  z^)  is  the  utility  function  over  the  i^ 
attribute  associated  with  state  h  and  action  k.  The  formulation  is  the 
result  of  several  key  simplifying  assumptions.  The  decision  maker  is 
assumed  to  be  risk  neutral,  so  that  he  is  indifferent  between  the  expecta¬ 
tion  across  a  set  of  uncertain  outcomes  and  the  uncertain  outcomes  themselves. 
This  allows  the  probabilities  to  be  entered  as  simple  coefficients.  Also, 
the  attributes  are  assumed  to  satisfy  additive  independence,  allowing  the 
linear  additive  form  of  aggregation.  Tests  for  compliance  with  these 
assumptions  can  be  found  in  V.  Winterfeldt  (1975)  or  Keeney  and  Raiffa  (1976). 

The  impact  of  a  message  or  item  of  data  is  to  change  the  probability  distri¬ 
bution  of  the  states  z^.  Once  the  message  is  received,  a  maximum  utility 
action  a*(y)  can  be  identified.  The  expected  utility  of  selecting  an 
information  source  S  then  becomes  (Emery,  1969): 

EU  (S)  =  l  l  P(zh)  P(y|zh)  u(a*(y) ,  zh) 

messages  states 

y  z 


(2-2) 


Here  u(a*(y),  z^)  is  the  utility  of  taking  action  a*(y)  given  that  state 
z^  occurs.  The  utility  function  is  again  multi-attributed,  but  for  simpli¬ 
city  u(a*(y),  z^)  is  portrayed  as  having  already  been  aggregated  across  the 
various  dimensions. 

This  type  of  analysis,  championed  by  such  researchers  as  Emery  (1969), 
Marshak  (1971)  and  Wendt  (1969),  is  suited  for  highly  structured  tasks. 

Not  only  must  the  possible  states,  messages,  actions,  and  outcomes  be 
specifiable,  but  the  prior  state  probabilities  and  the  conditional  proba¬ 
bilities  characterizing  the  information  system  must  be  derivable.  The 
sequence  of  decision  stages  can  be  depicted  using  a  decision  tree,  as 
shown  in  Figure  2-1.  The  tree  is  folded  back  by  associating  with  each 
possible  message  the  maximum  expected  utility  of  the  subsequent  actions. 

This  folding  back  represents  graphically  the  process  of  EU  maximization. 

The  favored  information  source  S  is  then  identified  by  comparing  the 
expectations  taken  over  all  possible  messages. 

2.3.2  Other  Methodologies.  A  number  of  other  techniques  have  also  been 
proposed  to  model  information  seeking  behavior.  Among  these  are  information 
theory  models,  (Whittemore  and  Yovits,  1973),  optimal  control  formulations, 
(Sheridan,  1976;  Rouse  and  Gopher,  1977),  queing  models,  (Rouse,  1975; 
Engstrom  and  Rouse,  1976)  and  information  integration  techniques  (Anderson 
and  Shanteau,  1970).  For  the  most  part,  these  techniques  demand  rigid 
problem  structuring  and  continuous  variables.  More  often,  the  conmunica- 
tion  decision  is  incompletely  defined  and  involves  choices  among  discrete 
rather  than  continuous  alternatives.  Thus  the  discrete  operators  used 
in  cue  regression  and  multi-attribute  utility  model s--matrices ,  difference 
operators,  and  detailed  parameter  enumerators--may  be  more  appropriate. 

The  interested  reader  is  directed  to  the  initial  technical  report  (Steeb, 
Chen  and  Freedy,  1977)  for  a  more  detailed  examination  of  these  approaches. 
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2.3.3  Information  Management  Functions.  The  major  information  manage¬ 
ment  functions  faced  by  the  operator  are  diagramed  schematically  in 
Figure  2-2.  The  information  available  consists  of  data  regarding  the  air¬ 
craft,  the  targets,  the  environment,  and  operator  and  system  capabilities. 
The  information  is  then  used  by  the  operator  to  perform  supervisory  control 
actions . 

The  information  and  control  choice  sequence  is  that  diagramed  with  the 
decision  tree  of  Figure  2-1.  This  diagram  is  repeated  with  labels  repre¬ 
sentative  of  tactical  airborne  operators  in  Figure  2-3.  The  multi-attribute 
utility  formulation  provides  a  useful  basis  for  structuring  both  the  infor¬ 
mation  and  control  decisions.  The  specific  steps  of  the  modeling  process 
are  outlined  in  Figure  2-4.  The  figure  shows  the  two  sides  of  the  modeling 
problem,  probability  estimation  and  utility  assessment.  The  upper  portion 
of  the  figure  details  the  processes  of  probability  estimation.  These 
include  delineation  of  the  possible  states  of  the  environment,  evaluating 
the  current  level  of  uncertainty  concerning  states,  selecting  information 
to  reduce  the  uncertainty,  and  revising  the  probability  estimates  in  light 
of  the  new  data.  The  lower  portion  of  the  figure  is  concerned  with  outcome 
evaluation  or  utility  estimation.  Here  the  levels  and  importance  weights 
for  each  dimension  of  consequence  are  determined. 

The  key  element  in  the  probability  estimation  sequence  is  the  information 
acquisition  stage  (enclosed  by  dotted  lines).  Figure  2-5  elaborates  this 
stage,  showing  the  steps  that  go  into  the  choice  of  information  and  the 
subsequent  incorporation  of  the  datum  into  the  situation  estimate.  The 
upper  portion  of  the  figure  deals  with  the  information  source  selection. 

The  characteristics  of  the  various  available  sources  are  determined  by 
observation  and  analysis.  This  estimation  of  the  characteristics  of  the 
information  sources  is  accomplished  by  successive  comparisons  of  messages 
received  and  subsequently  observed  states.  The  choice  of  information  source 
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is  then  made  according  to  the  potential  impact  of  the  information  on  the 
prior  probability  estimate.  Once  a  source  is  selected  and  a  datum  observed, 
the  information  is  incorporated  into  a  revised  situation  estimate  through 
Bayes'  rule: 


where 


PU^) 


p<*jl2h>  '  p<!h> 


<P<V 


P(yj)  ■  l  Uh)  '  P(*h) 


(2-3) 


p ( z h 1 y j )  is  the  probability  of  state  zh  being  present  given  that  message 
y.  was  received. 

J 


The  other  major  modeling  process  is  utility  assessment  or  outcome  evaluation. 
The  possible  combinations  of  actions  and  states  are  enumerated  off-line 
prior  to  a  mission.  The  problem  is  then  to  assign  consequence  levels  and 
importance  weights  along  a  predefined  set  of  dimensions.  Figure  2-6 
elaborates  this  process.  The  first  step  is  the  selection  of  an  independent, 
exhaustive,  and  predictive  attribute  set.  The  attributes  are  the  various 
constituent  aspects  of  the  consequences.  Each  combination  of  information, 
action  and  outcome  is  associated  with  a  set  of  attribute  levels.  This  is 
done  by  observation  and  adjustment,  just  as  in  the  determination  of  infor¬ 
mation  source  characteristics.  Scaling  procedures  are  applied  to  the  raw 
consequence  dimensions  to  arrive  at  normalized  values.  Each  attribute  is 
scaled  so  that  its  plausible  range  spans  zero  to  one.  These  processes 
result  in  a  specification  of  the  parameters  of  the  basic  multi-attribute 
formulation: 


E  [u(x)]s  =  l  P(zk) 
k=l 
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PROCESSES  INVOLVED  IN  OUTCOME  EVALUATION 


where 


E  [u(x)]  is  the  expected  utility  of  information  choice  s, 

P(zk)  is  the  probability  of  state  k  with  this  information  choice, 
is  the  importance  weight  for  attribute  i,  and 
is  the  level  of  attribute  i  associated  with  action  j  and  state  k. 


The  following  sections  will  develop  some  of  the  specifics  of  the  modeling 
cycle. 


2.3.4  Attribute  Development.  The  attributes  of  an  information  seeking 
decision  are  dimensions  of  consequence  that  are  common  to  all  tips  of  the 
decision  tree  (shown  earlier  in  Figure  2-1).  These  dimensions  may  include 
communications  costs,  equipment  losses,  goal  attainments,  future  effects, 
and  other  factors.  In  the  end,  the  constitutent  effects  will  be  weighted 
and  aggregated  together  to  arrive  at  an  overall  evaluation  of  an  outcome. 

The  actual  choice  of  the  attribute  set  is  extremely  important.  Dawes 
(1975)  states  that  the  choice  of  factors  to  include  is  probably  of  greater 
impact  than  the  determination  of  the  model  form.  Desirable  characteristics 
are  accessibility  for  measurement,  independence,  monotonicity  with  pre¬ 
ference,  completeness  of  the  set,  and  meaningfulness  for  feedback.  Mono¬ 
tonicity,  in  this  content,  implies  that  an  increase  in  the  attribute 
level  always  results  in  an  increase  in  preference.  If  the  attribute  levels 
are  monotonic,  a  simplification  is  possible.  Fisher  (1972)  and  Gardiner 
(1974)  note  that  a  straight  line  approximation  to  the  utility  function 
results  in  minor  losses  of  model  accuracy.  The  estimated  utility 
(ignoring  uncertainty  for  now)  is  then  a  weighted  linear  combination  of 
attribute  levels: 
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U(a2.  2h)  -  J  krxihk 

where  U(aT,  Is  the  utility  of  state  h  occurring  with  action  j,  k.  is 
the  importance  weight  for  attribute  i.  and  x.^k  is  the  level  of  attribute 
i  associated  with  state  h  and  action  k. 

Information  costs  may  comprise  attributes  of  special  note.  Often,  the 
benefits  of  an  Information  acquisition  are  simply  weighted  against  the 
costs  of  acquiring  the  information.  If  a  net  gain  is  anticipated,  acquisi¬ 
tion  of  the  information  is  considered  justified.  Often,  though,  the  costs 
themselves  are  multidimensional,  comprising  energy  costs,  time  delays, 
equipment  expenditures,  and  risks  of  detection.  The  scaling,  weighting, 
and  aggregating  of  these  costs  may  be  most  easily  performed  in  combination 
with  all  of  the  non-cost  attributes--tactical  gains,  political  impact,  etc. 
Then,  trade-offs  among  each  of  the  factors  may  be  performed  in  a  single, 
consistent  operation. 

A  candidate  set  of  attributes  might  contain  factors  from  five  areas: 

(1)  Communi cat  ions  Costs.  The  expenditures  associated  with  use 

of  the  information  sources.  These  may  include  requirements  of 
energy,  equipment,  and  operator  attention. 

(2)  Equipment  Attrition.  Consequences  concerning  the  integrity 
of  the  vehicle.  Included  are  fuel  expenditures,  system 
damage,  and  vehicle  loss. 

(3)  Objective  Attainment.  The  degree  of  accompl ishment  of  the 
mission  objectives.  Target  goals  may  bo  the  area  reconnoi tered. 
adversaries  dispatched,  and  political  impact  obtained. 

(4)  Dynamic  Effects.  The  future  consequences  resulting  from  the 
current  actions.  These  consequences  may  include  effects  on 
subsequent  action  choices,  availability  of  future  Information, 
and  changes  in  the  environment  resulting  from  the  action. 
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(5)  Subjective  Needs.  The  operator  may  have  propensities  for 

obtaining  (or  refusing)  information  beyond  that  called  for  bv 
the  above  factors.  These  preferences  reflect  the  needs  of 
task  continuity,  maintenance  of  load,  or  other  idiosyncratic 
factors. 

A  useful  consequence  set  might  contain  a  single  dimension  or  attribute  from 
each  of  these  categories.  In  fact,  five  attributes  appears  to  be  an  upper 
limit  to  the  number  of  factors  a  decision  maker  can  effectively  consider 
(V.  Winterfeldt,  1975).  If  several  factors  contribute  to  one  consequence 
dimension,  these  factors  should  be  combined  using  a  single  common  scale-- 
dollars,  ship-equivalents,  fuel  quantity,  etc. 

Each  of  the  attributes--communications  costs,  vehicle  losses,  etc., --must 
be  scaled  with  interval  properties  along  a  set  range.  The  least  desirable 
consequence  that  may  occur  is  assigned  a  level  of  zero  on  the  scale.  The 
most  desirable  consequence  is  assigned  a  level  of  one.  The  weighting 
factors  k j  should  also  be  normalized  so  that  the  overall  worst  combination 
of  factors  results  in  a  value  of  zero  and  the  overall  best  combination 
a  value  of  one. 

A  special  situation  occurs  with  probabilistic  attributes.  Assuming  risk 
neutrality,  probabilistic  consequences  may  be  computed  according  to  their 
expected  level.  For  example,  the  vehicle  loss  attribute  may  have  three 
possible  levels,  each  with  a  different  probability  of  occurrence.  The 
expected  value  is  computed  by  the  following  additive  expression: 
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where  the  parameters  are  defined  as  in  Equation  2-4.  Once  the  attributes 
are  defined  and  their  levels  are  determined,  the  aggregation  rule  must  be 
identified.  The  attributes— costs ,  losses,  delays,  future  impacts,  etc.,— 
may  combine  in  an  additive,  multiplicative,  or  more  complex  fashion  (see 
Keeney  and  Raiffa,  1975,  for  a  description  of  some  of  the  more  popular 
formulations).  For  the  work  here,  the  simple  additive  form,  exemplified 
by  Equation  2-5  appears  to  be  the  most  suitable.  The  additive  form  is 
robust,  intuitively  easy  to  understand,  and  simple.  Also,  the  linear  form 
of  the  additive  will  be  seen  to  be  amenable  to  estimation  by  pattern  recogni 
tion  techniques. 

2.3.5  Consequence  Level  Determination.  The  actual  level  of  each  of  the 
attributes  for  a  given  outcome  can  be  determined  by  mappings  between  pre¬ 
dictive  features  and  the  attributes.  Predictive  features  must  be  identified 
which  are  accessible  to  an  onboard  program  and  capable  of  determining  the 
consequence  levels.  Mappings  between  the  predictive  features  and  the 
attributes  are  either  pre-established  or  determined  by  observation  and 
adjustment. 

The  data  available  to  the  decision  program  are: 

(1)  Directly-sensed  information  concerning  the  environmental 
state  (weather,  terrain,  ECM,  target  track). 

(2)  The  vehicle  state  (velocity,  fuel,  autopilot  capability). 

(3)  The  information  system  characteristics  (capacity,  noice,  cost). 

(4)  Tactical  data  (technical  characteristics  of  own  and  enemy 
aircraft,  sensors  and  weapons;  information  about  the  operations 
area) . 

(5)  Action  alternatives  (control  responses,  weapon  deployment). 

(6)  Operator  capabilities  (attention,  load). 
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A  manageable  subject  of  these  features  must  be  determined.  The  consequence 
mapping  can  then  be  refined  by  comparison  of  the  predicted  and  actually 
observed  consequences.  The  mapping  can  be  developed  either  by  prior  defini¬ 
tion,  by  regression,  or  by  the  pattern  recognition  techniques  described  in 
the  coming  section. 

2.3.6  Attribute  Weight  Estimation.  The  policy  defining  factors  in  the 
model,  the  importance  weights  ,  are  parameters  suitable  for  either 
objective  or  subjective  estimation.  If  the  consequences  can  be  defined 
along  objective  scales  (dollars,  ship-equivalents,  etc.),  then  the  weights 
could  be  derived  by  analysis  and  input  prior  to  system  operation.  Unfor¬ 
tunately,  Felson  (1975)  states  that  only  in  a  few,  highly  structured  situa¬ 
tions  can  such  an  optimal  model  be  derived.  More  often,  the  operator's 
goal  structure,  expressed  as  importance  weights,  must  be  elicited  or 
inferred  and  then  incorporated  in  the  model.  There  are  a  number  of  advan¬ 
tages  to  such  subjective  estimation,  particularly  with  respect  to  allocation 
of  function.  By  incorporating  individualized  operator  weights  in  the  model, 
the  complex  evaluation  and  goal  direction  functions  remain  the  responsibility 
of  the  operator,  while  the  normative  aggregation  functions  are  assumed  by 
the  computer.  Also,  operator  acceptance  of  aiding  by  the  model  may  be 
increased  since  his  preferences  are  incorporated  in  the  machine  decisions. 

The  operator's  subjective  weights  may  be  defined  off-line  by  elicitation 
or  on-line  through  inference.  The  off-line  methods  include  direct  elicita¬ 
tion  of  preference,  decomposi tion  of  complex  gambles  into  hypothetical 
lotteries,  and  use  of  multi-variate  methods  to  analyze  binary  preference 
expressions.  These  techniques  are  accurate  and  reliable  in  many  circum¬ 
stances,  but  they  have  a  number  of  disadvantages  when  applied  to  operational 
systems.  Typically,  these  methods  require  two  separate  stages--assessment 
and  application.  Assessment  requires  an  interruption  of  the  task  and 
elicitation  of  responses  to  hypothetical  choices.  Problems  arise  with  such 
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procedures  since  the  operator's  judgments  may  not  transfer  to  the  actual 
situation;  the  decision  maker  may  not  be  able  to  accurately  verbalize  his 
preference  structure  (Macrimmon  and  Taylor,  1972);  and  the  judgments  made 
in  multi-dimensional  choices  are  typically  responses  to  non-general izable 
extreme  values  (Keeney  and  Sicherman,  1975). 

Estimation  techniques  relying  on  inference  from  in-task  behavior  may  be 
more  useful.  The  inference  techniques  can  be  based  on  non-parametric  forms 
of  pattern  recognition.  Here  a  model  of  decision  behavior  is  assumed  and 
the  parameters  of  the  model  are  then  fitted  by  observation  and  adjustment. 
Briefly,  the  technique  considers  the  decision  maker  to  respond  to  the 
characteri sties  of  the  various  alternatives  as  patterns,  classifying  them 
according  to  preference.  A  linear  discriminant  function  is  used  to  pre¬ 
dict  the  decision  maker's  choices,  and  when  amiss,  is  adjusted  using  error- 
correcting  procedures.  In  this  way,  no  preference  ratings  or  complex  hypo¬ 
thetical  judgments  are  required  of  the  operator. 

The  adaptive  nature  of  the  estimation  program  is  shown  in  Figure  2-7. 
Expected  consequence  vectors  associated  with  each  information  source  are 
input  to  the  model.  These  consequence  vectors  are  dotted  with  the  weight 
vector,  resulting  in  evaluations  along  a  single  utility  scale.  The  maximum 
utility  choice  is  determined  and  compared  with  the  operator's  actual  choice. 
If  a  discrepancy  occurs,  the  weight  vector  is  adjusted  according  to  the 
following  rule: 

k'  =  k  +  X(xc  -  xj  (2-7) 

where  J<‘  is  the  updated  weight  vector 
J<  is  the  previous  weight  vector 

\  is  an  adjustment  constant 

is  the  attribute  vector  of  the  chosen  alternative 
x^  is  the  mean  attribute  vector  of  all  alternatives  ranked  by 
the  model  above  the  chosen  alternative. 
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FIGURE  2-7. 

ADAPTIVE  ESTIMATION  PROCESS 


Ideally,  the  error  correction  moves  the  weight  vector  in  a  direction 
minimizing  subsequent  errors.  The  amount  of  movement  depends  on  x,  the 
adjustment  increment.  Nilsson,  1965,  describes  several  different  forms  of 
X  that  can  be  used  depending  on  the  combination  of  speed  and  smoothing 
desired. 

The  type  of  criteria  used  for  model  training  is  also  a  major  consideration. 
The  training  may  be  based  on  objective  outcomes  such  as  stock  market  con¬ 
sequences,  or  subjective  criteria  such  as  actual  operator  decisions,  or 
on  some  combination  of  subjective  and  objective  criteria.  The  approach 
based  on  both  objective  and  subjective  criteria  is  the  most  involved.  In 
many  situations,  an  occasional  indicator  of  objective  performance  is 
observable.  The  aircraft  may  be  lost,  the  target  attained,  or  some  number 
of  subgoals  accomplished.  In  this  way,  the  correctness  of  a  sequence  of 
actions  may  become  objectively  known.  The  utility  model  would  be  trained 
subjectively  prior  to  this  by  observation  of  the  operator's  choices.  If 
the  sequence  of  choices  led  to  an  objectively  favorable  outcome,  the 
trained  parameter  set  would  be  retained.  If  the  outcome  was  unfavorable, 
the  parameter  set  would  be  returned  to  the  levels  present  prior  to  the 
sequence  of  decisions.  In  this  way,  objective  criteria  would  guide  overall 
training,  but  the  explicit  decision-by-decision  policy  for  information 
management  would  be  subjectively  derived. 

Of  course,  the  adaptive  techniques  of  estimation  described  above  are 
warranted  only  if  repetitive  decisions  are  available  for  training  and  if 
the  differential  weighting  of  attributes  is  important.  In  cases  where  only 
a  few  non-repeti ti ve  decisions  will  be  made,  off-line  estimates  of  the 
weights  k^  are  favored.  Here,  techniques  such  as  direct  estimates,  hypo¬ 
thetical  lotteries,  or  multiple  regression  are  used  for  estimation  prior 
to  the  mission.  It  is  assumed  with  these  techniques  that  the  system 
requirements  will  not  change  after  the  estimates. 
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Questions  concerning  the  importance  of  differential  weighting  are  more 
basic.  Unit  weighting  schemes  (in  which  all  weights  ki  are  set  equal  to 
1.0)  have  been  found  to  be  quite  effective  in  certain  circumstances.  Errors 
in  the  model  form,  positive  correlations  between  variables,  and  small  sample 
sizes  all  reduce  the  predictive  capabilities  of  differential  weights  com¬ 
pared  to  unit  weight  (Einhorn  and  Hogarth,  1975).  Essentially,  the  more 
precise  and  parsimonious  the  model,  the  more  important  differential  weights 
are. 

Unit  weighting  schemes  are  expected  to  see  only  minor  application  in  aiding 
advanced  aircraft  operations.  Careful  selection  of  attributes  minimizes 
intercorrelations  between  variables,  and  the  correlations  that  do  occur 
should  tend  to  be  negative.  For  example,  in  most  cases  costly  information 
is  generally  more  informative  than  inexpensive  information,  and  equipment 
attrition  tends  to  be  negatively  correlated  with  goal  attainment.  These 
circumstances  favor  inferred  weight  models.  Unit  weighting  schemes  should 
primarily  be  useful  as  starting  points  for  estimation,  or  as  strategies 
for  situations  in  which  a  great  deal  of  noise  is  present. 

2.3.7  Probability  Estimation.  The  major  probability  parameters  requiring 
estimation  are  the  prior  probabilities  P(z)  and  the  conditional  probabilities 
P(y|z).  The  priors  are  the  probabilities  of  state  z  in  a  particular  situa¬ 
tion.  The  conditional  probabilities  deal  with  the  likelihood  of  receipt  of 
message  y  if  state  z  is  present.  Both  of  these  forms  of  probabilities  can 
be  estimated  from  frequency  counts. 

A  second  area  of  uncertainty  concerns  the  consequence  levels  associated 
with  a  given  message  and  state.  These  are  the  performance  probabilities 
and  are  derived  from  stored  data:  detection  range,  target  hardness,  per¬ 
sonnel  performance,  system  reliability,  guidance  system  accuracy,  etc.  The 
probability  of  outcome  given  the  message  received  can  be  computed  for  each 
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set  of  actions.  Comparison  of  the  messages  received,  actions  taken  and 
the  consequences  subsequently  observed  provide  the  necessary  data. 

2.3.8  Level  of  Detail .  The  level  of  detail  handled  by  the  model  is 
basically  a  problem  of  efficiency  of  categorization.  Choices  of  the  fine¬ 
ness  of  distinction  of  information  modes,  messages,  and  states  involves  a 
trade-off  of  the  model  complexity  (and  processing  time)  against  degree  of 
specification  of  aiding.  For  example,  the  information  available  may  be 
classified  by  content  element  (threat  identify,  weather  formation,  mal¬ 
function  location,  terrain  characteristics,  etc.)  or  by  source  (radar, 
infra-red  sensor,  video,  tactical  data  base,  radio,  etc.).  Similarly,  the 
messages  and  states  may  be  represented  in  the  model  as  specific  details  of 
threat  position,  course,  speed  and  identification,  or  more  globally  as 
specific  threat  presence  or  absence.  Of  course,  separate  prior  and  condi¬ 
tional  probability  estimates  must  be  maintained  for  each  state  represented 
in  the  model . 

Much  of  the  question  of  level  of  detail  was  to  do  with  the  concept  of  pay¬ 
off  relevance,  a  term  introduced  by  Marschak  (1963).  The  partitioning  of 
the  information  space  must  result  in  differences  in  (1)  the  existing  repre¬ 
sentation  of  the  decision  situation,  (2)  the  actual  decisions  made,  and 
(3)  the  utility  resulting  from  the  changed  decisions.  Information  may  be 
ineffective  in  changing  the  situational  representation  and  resulting 
decisions  because  the  data  is  too  coarse  or  too  fine.  Information  that  is 
too  coarse  fails  to  distinguish  between  effectively  different  states  of 
nature  for  at  least  one  of  the  alternative  actions.  Information  that  is 
too  fine  differentiates  between  states  having  identifical  payoffs  for  all 
actions.  Effective  information--data  that  is  not  too  fine  or  too  coarse-- 
is  termed  by  Marschak  to  be  payoff  relevant.  In  the  experimental  applica¬ 
tion  of  the  model ,  attempts  were  made  to  structure  the  model  so  that  the 
cost  of  the  chosen  level  of  detail  is  commensurate  with  the  benefits. 
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Aiding  in  Advanced  Aircraft  Operations 


2.4.1  Forms  of  Aiding.  It  was  noted  earlier  that  thr  e  major  forms  of 
aiding  are  possible  once  the  operator's  decision  policy  is  estimated: 

(1)  system  evaluation  and  design,  (2)  automated  management  of  information, 
and  (3)  operator  training.  This  section  will  describe  in  more  detail  the 
potential  of  model-based  aiding  for  tactical  airborne  systems.  Recounting 
briefly,  system  evaluation  and  design  entails  the  determination  of  each 
system  component's  contribution  to  information  value  (satisfaction  of 
operator  needs).  Specification  of  system  configuration  commensurate  with 
task  demands  and  individual  operator  needs  is  then  possible.  Automated 
management  of  information  is  the  moment- by-moment  control  of  display  con¬ 
tent  and  format  by  selection  among  available  information.  Operator  training, 
finally,  involves  the  use  of  policy  feedback  to  train  the  operator  to  make 
effective  information  and  control  decisions.  The  first  two  of  these  uses 
of  the  decision  model,  system  evaluation  and  automated  information  manage¬ 
ment,  will  be  described  in  greater  detail  in  the  following  sections.  Model- 
based  training  is  presently  outside  the  scope  of  this  work. 

2.4.2  System  Evaluation.  Two  types  of  evaluation  are  possible  using  the 
information  extracted  from  the  information  value  model:  direct  contribution 
and  marginal  contribution.  Each  of  these  evaluation  measures  is  described 
below. 

Direct  Contribution.  This  is  the  user-specific  value  of  a  given  information 
source  in  a  given  task  situation.  As  such,  it  is  a  simple  aggregation  of 
components,  weighted  by  the  user's  policy: 

info  value  jks  -  J  k(j  xjk  (2-8) 

attributes 

i 
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where  info  value  is  the  aggregate  value  of  source  s  to  user  j  in 
situation  k,  k..  is  the  importance  weight  of  attribute  i  to  user  j;  and 

__  » J 

x..  is  the  mean  level  of  attribute  i  in  situation  k.  This  formulation  is 

I  <J 

useful  when  each  information  source  contributes  to  a  different  task-- 
threat  detection,  navigation,  etc.  The  direct  contribution  measure  does  not 
deal  with  information  sources  having  overlapping  function. 

Marginal  Contribution.  In  a  group  of  information  sources  with  overlapping 
function,  the  information  value  of  one  source  can  be  calculated  with  the 
following  expression: 


information  value  ^  =  £  k . ^ x ^ ^ 

attributes 

i 


max 
over 

remaining 
sources  i 


^  k.  .x.. 
L  lj  lk 


(2-9) 


This  is  the  incremental  value  of  a  source  over  the  next  most  highly  valued 
source.  The  summation  of  all  positive  contributions  for  a  given  source 
indicate  the  source's  value  in  the  particular  task  situation. 

Using  either  the  direct  or  marginal  measure  of  information  value,  the 
mission  value  of  an  information  source  can  be  calculated  as  the  summation 
across  the  probability  distribution  of  task  situations. 

(2-10) 

info  value  jS  =  [  prob  (situation  k)  *  information  value 
k 

This  provides  an  overall,  user-specific  index  of  information  value. 

2.4.3  Automated  Management  of  Information.  The  information  value  model 
described  in  section  2.3  can  be  used  directly  for  management  of  information. 
The  multi-attribute  utility  model  represents  the  policy  of  the  specific 
user,  it  has  access  to  the  factors  characterizing  each  information  choice. 
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and  it  can  be  linked  to  the  onboard  information  control  system.  The  model 
can  thus  be  configured  to  automatically  scan  the  available  information 
sources,  select  the  immediately  most  useful  source,  and  display  it  to  the 
operator.  The  following  sections  describe  the  initial  experimental  appli¬ 
cations  of  these  aiding  programs. 

2.5  Experimental  Validation 

Evidence  for  the  usefulness  of  the  multi -attribute  utility  formulation 
and  adaptive  estimation  programs  was  obtained  during  the  initial  year  of 
the  program  (Steeb,  Chen  and  Freedy,  1977).  A  simulation  resembling  con¬ 
trol  of  a  remotely  piloted  vehicle  (RPV)  was  used  in  this  study.  Individual 
subjects  navigated  the  RPV  through  a  changing  hazardous  environment.  In 
doing  so,  the  operators  selected  different  combinations  of  information  and 
control  allocation.  The  adaptive  model  was  found  to  be  significantly  more 
predictive  of  subject's  behavior  than  either  a  constant,  unity  weight  model 
or  an  off-line  method  of  weight  estimation.  Also,  the  model  was  found  to 
be  useful  in  identifying  different  decision  policies  or  styles. 

Use  of  the  adaptive  and  off-line  models  to  make  choice  recommendations  to 
the  operators  had  mixed  results.  The  differences  in  task  performance  noted 
between  the  recommendation-aided  and  unaided  conditions  did  not  reach 
significance,  although  those  who  followed  the  recommendations  most  closely 
achieved  the  highest  scores.  Also,  the  adaptive  model  was  found  to  be 
useful  in  identifying  strategies  which  led  to  superior  performance. 

The  second  year  effort  (Steeb,  Davis,  Alperovitch  and  Freedy,  1978)  built 
on  the  findings  of  the  initial  work  by  investigating  the  usefulness  of  the 
information  value  models  for  automating  the  presentation  of  information  and 
for  evaluating  the  effectiveness  of  information  display  configurations. 

The  program  also  re-directed  the  application  area  from  one  of  remotely 
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piloted  vehicle  supervision  to  one  of  information  selection  in  advanced 
aircraft.  A  simulation  based  on  multiple  threat  intercept  operations  in 
advanced  aircraft  was  developed.  Additional  factors  for  time  stress  and 
an  expanded  information  and  action  set  were  included  in  the  new  decision 
model . 

The  primary  focus  of  this  second  series  of  experiments  was  to  test  the 
effectiveness  of  the  adaptive  decision  model  for  information  management  and 
information  svstem  evaluation.  Subjects  (12  in  the  study)  were  required  to 
select  from  a  variety  of  forms  of  information  regarding  multiple,  uncertain 
threats  and  to  take  aggressive  or  avoidance  actions  in  response.  The 
information  options  differed  in  cost,  time  delays,  threat  discrimination 
and  enemy  detection.  The  operators  experienced  a  sequence  of  decisions 
organized  into  mission  phases.  Comparisons  were  made  between  (1)  automated 
information  management  based  on  the  adaptive  model  described  earlier,  (2) 
automated  information  management  based  on  information  seeking  strategies 
elicited  directly  from  the  operator,  and  (3)  manual  information  selection. 
Each  subject  experienced  sessions  of  each  of  these  conditions  under  low 
and  high  speed  stress.  Information  management  using  either  automated  form 
was  found  to  result  in  improved  task  performance  over  manual  selection. 

The  improvement  with  aiding  was  enhanced  in  situations  of  high-speed  stress. 
The  performance  score  improvement  with  adaptively-based  management  over 
manual  selection  changed  from  30  percent  improvement  in  the  low-speed 
stress  conditions  to  60  percent  improvement  in  the  high-speed  stress  condi¬ 
tions.  Finally,  the  adaptive  technique  was  found  to  be  superior  to  direct 
policy  elicitation,  both  for  automated  information  management  and  as  a 
basis  for  information  system  evaluation. 

The  following  sections  extend  these  results  both  experimentally  and 
analytically  to  the  more  complex  and  realistic  situations  of  limited 
resources,  continued  sampling  of  information,  and  variable  operator  loading. 
Also,  the  full  process  of  information  acquisition,  processing,  formating  and 
display  will  be  considered. 
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3.  EXPANDED  SYSTEM  DEVELOPMENT 


The  preceding  sections  have  described  how  an  adaptive  computer  program 
organized  around  a  multi-attribute  decision  model  can  be  used  for  modeling 
and  even  automating  the  information  selection  processes.  Up  to  this 
point,  the  model  has  by  necessity  been  developed  in  a  relatively,  simple 
context.  While  the  analysis  has  addressed  problems  of  multiple  criteria, 
probabilistic  consequences  and  time-varying  behavior,  a  variety  of  other 
factors  must  be  considered.  For  example,  the  emphasis  has  been  on  the 
choice  of  information  for  transmission  and  presentation.  It  is  necessary 
to  expand  the  domain  of  the  information  evaluation  methodology  to 
explicitly  represent  the  complete  sequence  of  sensor  selection  and 
adjustment,  data  encoding  and  transmission,  and  information  processing 
and  display.  Also,  the  influences  on  model  performance  of  operator 
loading,  continued  sampling  of  information,  and  limited  resources  have 
not  been  determined.  These  important  questions  will  be  dealt  with 
analytically  in  the  following  sections.  Experimental  studies  of  the 
effects  of  limited  resources  and  operator  loading  on  the  information 
value  models  will  be  described  in  the  coming  chapter. 

3.1  Limited  Resources  and  the  Use  of  Cost-Benefit  Decision  Criteria 

Tactical  Airborne  Operations  are  often  subject  to  constraints  on  certain 
resources.  Fuel,  weapon  stores,  and  time  aloft  may  be  constrained  by 
absolute  limitations.  The  previously  described  linear  additive  MAU 
formulation  treats  each  of  these  factors  as  additive  dimensions  of 
value.  For  effective  operation,  however,  the  flight  officer  would  be 
expected  to  choose  actions  which  maximize  the  benefits  per  cost  expended 
until  the  resource  is  exhausted.  This  process  is  exactly  that  of 
cost-benefit  analysis.  Here  the  benefit-to-cost  ratios  u(x)/c  are 
calculated  for  each  alternative,  and  the  actions  with  the  highest  ratios 


are  chosen  (Edwards  and  Guttentag,  1975;  Fischoff,  1977).  For  ratio 
scale  properties  to  hold,  benefits  must  be  calculated  with  respect  to 
a  zero  cost  option. 

The  resulting  information  selection  criterion  is  somewhat  more  complex 
than  the  additive  MAU  formulation.  The  benefit/cost  ratio  of  a  given 
alternative  is: 

V  k.  A.. 

*•  1  ik 

Benefit/Cost  ratio  (option  .)  =- -  (3-1) 

V  C.. 

L  i  ik 

i 

Where  k.  is  the  importance  weight  of  (benefit)  attribute  i,  A^  is  the 
level  of  attribute  i  associated  with  option  k,  k^  is  the  importance 
weight  of  (cost)  attribute  t,  and  Cik  is  the  level  of  attribute  t  asso¬ 
ciated  with  option  k. 

The  cost-benefit  formulation  is  amenable  to  adaptive  estimation  in  the 
same  manner  as  the  additive  MAU  criterion  of  equation  2-4.  The  attribute 
levels  divided  by  the  option  cost  (A^/C^)  replace  the  A^  in  Figure  2-7. 
The  comparison  of  recommended  and  actually  chosen  options  takes  place  as 
before.  However,  a  modification  of  the  adjustment  rule  is  necessary, 
since  the  adjustment  procedure  does  not  produce  a  change  in  the  importance 
of  cost.  In  fact,  if  two  options  differ  widely  in  cost,  the  chances  are 
that  the  lower  cost  option  will  be  chosen  regardless  of  other  variables. 
The  adjustment  would  then  have  only  a  50  per  cent  change  of  moving  in  the 
correct  direction.  The  highest  probability  of  correct  adjustment  would 
occur  when  the  option  costs  are  equal.  Then  all  variance  (and  adjustment) 
takes  place  with  the  weighted  factors.  An  adjustment  procedure  appro¬ 
priate  to  this  situation  is  the  following: 
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K1  =K+Aa(x-  X  ) 
'  c  m 


where  a  = 


C  +  C 
c  ^p 


(3-2) 


Cc  is  the  cost  of  the  chosen  alternative  and  the  remaining  variables  are 
defined  as  in  equation  2-7. 


In  this  expression,  a  takes  the  range  of  0  <  a  <  1  and  is  inversely  pro¬ 
portional  to  the  difference  in  costs  between  the  chosen  (Cc)  and  predicted 
(Cp)  options.  Adjustments  are  then  largest  when  the  cost  difference  is 
smal 1 . 


The  cost-benefit  formulation  differs  from  the  additive  MAU  form  in  the 
combination  rule  of  the  factors  and  in  the  adjustment  procedure.  The  two 
formulations  are  compared  in  the  experimental  study  (Section  4)  to  see 
if  the  potential  performance  advantages  of  the  cost-benefit  form  outweighs 
the  additional  complexity  and  training  time  of  the  purely  additive  model. 

3.2  Continued  Sampling  of  Information 

Up  to  this  point,  the  analysis  of  the  information  seeking  task  in  advanced 
aircraft  operations  has  dealt  exclusively  with  single-sample  situations-- 
tasks  were  the  possible  information  options  are  scanned,  a  single  option 
or  set  of  options  selected  and  observed,  and  an  action  taken.  The  more 
realistic  but  also  much  more  complex  case  is  that  of  sequentially 
sampling  different  sources  until  some  confidence  level  is  achieved  prior 
to  action  execution.  This  is  termed  continued  sampling  of  information. 
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Continued  sampling  occurs  frequently  in  the  cockpit.  The  pilot  receives 
a  radio  communication  regarding  possible  threats.  The  radar  is  set  to 
wide  envelope  scanning.  Detection  of  a  threat  then  leads  to  use  of 
narrow  envelope  radar  or  infrared  signature  analysis  until  the  threat 
identity  is  established  with  sufficient  confidence  to  take  action. 

The  above  scenario  fits  nicely  into  the  modeling  paradigm  of  optional 
stopping.  Optional  stopping  occurs  when  the  decision  maker  must  choose 
between  accepting  a  currently  available  action  or  continuing  to  sample 
further  information  to  achieve  greater  certainty.  A  wealth  of  experimental 
information  has  been  accumulated  in  this  context.  In  medical  diagnosis, 
for  example,  Pitz  (1968)  found  that  the  cost  of  each  diagnostic  test  is 
compared  with  the  expected  gain  which  it  provides.  Descriptively,  it 
is  found  that  people  purchase  information  until  posterior  odds  in  favor 
of  the  most  likely  hypothesis  reach  some  critical  value.  The  critical 
value  decreases  and  the  sample  size  increases.  Levine  and  Samet  (1973) 
found  that  subjects  purchase  more  information  prior  to  a  decision  where 
the  information  was  (a)  higher  reliability,  (b)  higher  update  frequency, 
and  (c)  increased  degree  of  conflict.  Also  Snapper  and  Peterson  (1971) 
found  that  the  degree  of  diagnosticity  also  influenced  information 
purchasing,  along  with  cost  and  prior  probability. 

Unfortunately,  the  optional  stopping  behavior  has  been  successfully 
modeled  only  in  the  two  hypothesis  case  (Edwards,  1965;  Slovic,  Fishburn 
and  Lichtenstein,  1977).  The  advanced  aircraft  situation  and  indeed 
even  the  current  task  simulation  exhibits  a  large  number  of  distinct 
hypotheses.  Each  combination  of  threat  type  and  location  represents 
a  distinct  state. 

The  problem  can  be  seen  in  Figure  3-1,  a  modification  of  the  decision 
tree  of  Figure  2-3.  Each  possible  information  sampling  can  be 
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FIGURE  3-1. 

CONTINUED  SAMPLING  DECISION  TREE 
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represented  as  an  action  with  definable  consequences  (costs,  delays, 
etc.).  Subsequent  information  samplings  and  actions  along  this  branch 
are  contingent  on  the  information  received.  In  order  for  the  tree  to 
be  useful,  all  of  the  possible  sequences  of  messages,  actions,  and 
events  must  be  definable. 

A  possible  reduction  of  the  effort  is  possible  through  integration 
theory  (Anderson,  1965;  Shanteau,  1970).  Integration  theory  asserts 
that  the  step-by-step  buildup  of  judgment  (estimated  likelihood  of 
avoidance,  for  instance)  may  be  expressed  by: 

Rk '  8k-i  *  “k  <VRk-i>  l2-3> 

Where  R.  is  the  judgment  after  receipt  of  the  k^  item  of  information, 
k  t.h 

is  the  judgment  prior,  Sk  is  the  scale  value  of  the  kin  item,  and 

Wk  is  a  change  parameter  that  measures  the  influence  of  the  kth  item. 

This  formulation  can  be  a  useful  descriptive  measure  in  which  the 
parameters  are  derived  by  regression  from  observed  behavior.  Again, 
however,  problems  develop  since  the  use  of  a  constant  assumes  that 
each  item  of  information  reduces  the  uncertainty  by  a  set  amount. 

Some  combination  of  the  Bayesian  approach  and  the  integration  theory 
method  appears  useful.  The  possible  information  acquisition  choices 
may  be  categorized  into  three  classes,  according  to  the  form  of  model: 

(1)  Bayesian  -  Only  a  few  distinctly  different  messages  are 
possible,  precise  estimation  of  probabilities  is 
necessary,  and  the  prior  and  conditional  probabilities 
are  available  with  some  accuracy. 
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(2)  Integration  Theory  -  A  given  information  source  provides 
a  wide  variety  of  messages  but  the  message  influence  is 
constant  in  terms  of  diagnosticity. 


(3)  Unmodeled  -  The  information  choice  is  too  poorly  defined 
to  allow  either  of  the  above  forms.  In  this  case,  an 
overall  judgment  of  the  value  of  the  information  may  be 
assigned. 

The  above  forms  may  be  used  at  different  parts  of  the  decision  tree 
faced  by  the  operator.  In  all  of  the  above  cases,  a  posterior 
estimate  of  the  state  is  necessary  from  the  previous  information 
sampling.  This  posterior  estimate  may  come  from  an  updata  calculation 
such  as  from  equation  3-3  or  from  a  subjective  estimate  or  threshold 
response. 


Process  Model  of  Information  Seekinc 


The  information  seeking  and  decision  making  models  described  up  to 
this  point  have  dealt  primarily  with  the  processes  of  evaluation  and 
selection  of  an  information  source.  Implicit  in  this  analysis  are 
considerations  of  the  characteristics  of  the  information  sources,  the 
means  of  transmision,  the  type  of  display  presentation  and  formatting, 
and  the  subsequent  action  decisions.  In  order  to  represent  (and  optimize) 
relations  between  each  of  these  stages,  a  complete  process  model  is 
needed. 


A  generalized  organization  of  the  process  model  in  block  form  is  shown 
in  Figure  3-2.  Inputs  from  each  of  the  information  sources  (radar,  fault 
detection,  etc.)  are  processed  in  preparation  for  display.  The  proces¬ 
sing  results  in  determination  of  attribute  levels  for  each  input,  and 


these  attribute  levels  are  routed  to  the  formatting  program.  The  format¬ 
ting  program  determines  the  impact  of  candidate  display  formats  on  a 
subset  of  the  information  attributes--delay ,  effort,  and  discrimination, 
for  example.  The  full  set  of  attributes  for  each  option  and  format  is 
then  transferred  to  the  information  value  model  for  evaluation  and 
selection.  The  model  determines  the  most  effective  type  of  information 
and  formatting  in  the  situation  at  hand,  and  manages  the  display  accordingly. 

The  portions  of  the  information  system  described  up  to  this  point  have  been 
operating  in  a  deterministic  input-output  mode.  The  remaining  portions 
of  the  flow  chart  show  the  adaptive  elements  of  the  program.  The  first 
feedback  loop  is  found  between  the  operator  and  the  information  value 
model.  After  observing  the  selected  information,  the  operator  may  use 
it,  dispense  with  it,  or  request  other  information.  If  the  operator 
overrides  the  information  choice,  this  is  an  input  to  that  program, 
resulting  in  the  operation  of  the  error  correcting  adjustment  procedures 
described  in  Section  2-3  and  3-1.  A  probability  adjustment  feedback  is 
seen  between  the  environment  and  the  probability  model.  The  actions  and 
subsequent  outcomes  are  fed  into  the  probability  model  and  probability 
estimates  are  built  up  by  observation  and  adjustment.  For  each  mission 
situation,  variations  in  decision  strategy  and  event  likelihoods  are 
input  from  a  mission  situation  data  base.  The  final  form  of  feedback  is 
the  adjustment  of  the  information  sources  using  the  information  value 
model.  An  optimization  program  based  on  gradient  search  can  determine 
the  sensor  characteristics--radar  scan  volume,  power,  direction,  etc. 
needed  to  maximize  the  processed  information  attributes. 

Implementation  of  the  above  system  requires  an  elaborate  model  specifying 
the  influences  of  source  type,  processing  characteristics,  format 
specification,  and  situation  demands  on  the  information  contribution. 

The  process  may  be  viewed  as  a  series  of  transitions  between  each  of  the 
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elements  in  the  information  cycle.  The  information  begins  the  cycle  with 
a  set  of  characteristics,  which  are  degraded  or  enhanced  with  each  element. 
For  example,  radar  may  pick  up  a  set  of  threats.  Processing  may  identify 
the  threat  types  and  determine  their  course.  Formatting  the  information 
graphically  may  reduce  the  location  accuracy,  add  to  the  noise  present, 
and  result  in  a  display  time  delay.  The  mission  situation,  finally,  will 
specify  the  importance  of  rapid  offensive  action  or  avoidance. 

Each  of  the  transitions  described  above  may  be  represented  using  matrix 
methods.  For  example,  the  influence  of  display  format  on  information 
attributes  may  be  described  using  a  matrix  of  the  type  of  Figure  3-3. 

The  information  attributes  correspond  to  the  impact  of  the  information 
on  the  given  dimension  of  value.  The  elements  of  the  matrix  reflect  the 
strength  of  communication  of  each  dimension  or  attribute  by  the  given 
display  format.  For  example,  an  alphanumeric  display  may  preserve  the 
precision  level  of  a  pitch  or  roll  reading,  but  may  result  in  cognitive 
loading  and  delay.  A  symbolic  representation  on  the  other  hand,  may 
degrade  the  precision  level  but  tends  to  be  rapidly  assimilated.  In  sum, 

the  matrix  techniques  provide  a  means  of  expressing  the  influence  of  each 

type  of  processing  on  the  information  characterisitcs. 

A  linking  together  of  matrices  to  represent  much  of  the  information 

management  process  is  shown  in  Figure  3-4.  Tracing  the  path  between 

processes,  it  can  be  seen  that  the  first  matrix  relates  information 
sources  to  attributes.  Essentially,  the  sources  are  decomposed  into 
costs,  delays,  and  other  consequences  independent  of  the  display  format. 
This  is  equivalent  to  the  mapping  between  mission  situation  and  decision 
attributes  discussed  in  Section  2.3.5.  The  matrix  elements  are  determined 
by  subjective  estimation  or  by  observation  and  adjustment.  The  second 
matrix,  relating  information  attributes  to  display  format,  shows  how  the 
attributes  are  affected  by  the  display  format.  The  final  matrix  is 
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FIGURE  3-3. 

INFORMATION  TRANSFER  MATRIX 


the  weighting  of  the  attributes  according  to  importance  by  mission  situation. 
The  adaptive  or  direct  elicitation  procedures  described  in  Section  2.3 
provide  estimates  of  these  elements.  Should  additional  dimensions  of 
transmission  mode  or  decoding  form  be  present,  matrices  may  be  defined 
for  these  also. 

Once  the  individual  matrices  are  defined  and  filled  in,  the  best  set  of 
information  management  options  may  be  selected  through  a  multiplication 
of  the  matrices.  The  source,  processing,  and  format  with  the  greatest 
net  contribution  can  then  be  identified.  Conversely,  the  sources  may  be 
adjusted  through  a  search  procedure  to  maximize  the  aggregate  contribution 
for  each  set  of  options.  The  matrix  representation  also  allows  for  various 
forms  of  sensitivity  analysis.  Each  aspect  of  source  characteristic, 
processing  mode,  or  display  format  may  be  varied  to  determine  its  impact. 


Operator  Loadinc 


The  task  loading  on  the  flight  officer  is  expected  to  vary  widely  during 
a  mission.  The  loading  may  be  a  result  of  decision  complexity,  the  speed 
or  frequency  of  decision-making,  or  the  number  of  tasks  (secondary  task 
loading)  demanded  of  the  operator.  The  result  of  the  excessive  loading 
may  be  degraded  decision  making,  "narrowing"  to  a  subset  of  factors, 
inadequate  response  time,  or  the  ignoring  of  certain  processes.  These 
suboptimal  behaviors  may  be  averted  through  use  of  several  different 
forms  of  aiding: 


(1)  Reducing  information  throughput  at  heavily  loaded  times 
through  use  of  a  variable  utility  threshold. 

(2)  Changing  the  additive  importance  of  time  related  factors 
(delay,  speed,  etc.)  as  the  situation  demands  change. 

(3)  Providing  greater  amounts  of  unburdening  through  auto¬ 
mation  when  the  task  complexity  is  excessive. 
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A  necessary  precursor  to  implementation  of  any  of  these  forms  of  aiding 
is  the  development  of  a  methodology  for  measurement  of  immediate  workload. 
A  useful  concept  for  the  assessment  of  workload  has  been  proposed  by 
Johns  (1973).  He  divided  the  broad  area  of  human  operator  load  into  three 
functionally  related  attributes: 

(1)  Input  load  -  factor  or  events  external  to  the  operator. 

(2)  Operator  effect  -  internal  to  the  operator. 

(3)  Performance  -  data  outputs  generated  by  the  operator  which 
serve  as  inputs  to  system  components. 

Some  measures  of  performance  that  fall  into  these  categories  are  listed 
in  Figure  3-5  (after  Johannsen,  1976). 

Measures  of  operator  load  are  of  great  concern  and  traditionally  regarded 
as  operator  workload.  In  time  line  analysis  (e.g..  Greening,  1978),  the 
execution  times  of  all  subtask  elements  are  assessed.  It  is  assumed  that 
available  time  margins  or  expected  time  stress  correlate  with  level  of 
effort  expended  by  the  operator. 

In  information  processing  studies,  the  operator's  channel  is  regarded  as 
an  information  processing  element  with  a  fixed,  limited  channel  capacity. 
Three  major  approaches  are  considered: 

(1)  Secondary  or  loading  task  -  to  measure  spare  mental  capacity. 

(2)  Control  theoretic  measure  -  time  or  frequency  domain 
(amplitude  or  power  spectrum  density  measures)  of  human 
controller  models  (Baron  and  Levi  son.  1975). 

(3)  Information  Theory. 
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ATTRIBUTES  OF  OPERATOR  WORKLOAD 


One  major  effort  of  man-inachine  system  modeling  is  to  provide  an  accurate 
model  of  human  operator  with  predicted  operator  effort  and  performance  as 
a  function  of  the  given  task  situation.  A  major  limitation  of  this 
approach  is  due  to  the  fact  that  human  behavior  is  not  context-free,  i.e., 
it  strongly  reflects  the  task  environment.  If  one  looks  at  flight  control 
systems  that  include  control  with  respect  to  continuous  events,  then  one 
can  subsume  most  analytical  models  within  the  very  specialized  category 
of  control  theory. 

There  appears  to  be  a  level  of  load  that  maximizes  human  information 
processing  performance.  If  the  human’s  workload  is  too  high,  overload 
and  degraded  performance  occurs  (Van  Gigch,  1971).  Human  performance 
also  degrades  when  the  human  is  underloaded  (Van  Cott  and  Warrick,  1972). 
However,  the  human  does  not  usually  maintain  the  optimal  level.  A 
recent  study  (Hart,  1979)  showed  that  the  pilots  wanted  excessive 
amounts  of  information  onboard,  including  cockpit  displays  of  traffic 
information;  and  as  suggested  in  a  review  of  literature  (Samet,  et  al , 
1976),  the  operator  typically  desires  more  information  than  can  be  mean¬ 
ingfully  used. 

With  the  use  of  model-based  aiding  of  information  selection,  it  is  possible 
to  adaptively  adjust  the  pilot's  information  load.  The  computer  can 
observe  the  operator's  actions  and,  using  a  model  of  human  behavior, 
decide  on  the  appropriate  level  of  human  workload.  The  computer  can 
then  adapt  the  information  presentation  to  the  model. 

From  an  information  system  point  of  view,  a  simplified  model  is  to  view 
the  pilot's  information  and  decision  load  as  reduction  of  information 
content: 

•iHi  =  [H^ (x)  -  H^y)]  At;  i  =  l,..n 
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Where  H(x)  and  H(y)  are  information  content  rates  (in  bits/sec)  that  the 
operator  receives  from,  and  provides  to  the  system.  Four  variables  that 
affect  the  information  load  are:  (1)  the  information,  rate  (speed  stress), 

H ^ ( x ) ,  (2)  the  information  complexity,  (x ) -H^ (y ) ,  (3)  the  number  of 
processes  (subtasks),  n  and  (4)  the  level  of  interaction  among  processes; 
for  processes  i  and  j,  this  can  be  represented  by  Cov(H.(y),  H.(x)), 

’  J 

cov(H.(y),  H .  (x) ) .  These  variables  of  input  load  could  serve  as  good 

J  ■ 

criterion  for  load  adjustment  provided  that  the  input  load  and  perfor¬ 
mance  relationship  for  specific  tasks  are  established. 

An  alternative  in  the  experimental  setting  is  to  use  a  secondary  task  as 
a  measure  of  pilot  information  load.  Here  the  operator  is  considered  as 
a  single-channel  information  processing  system.  This  single  channel 
concept  can  be  shown  to  be  realistic  when  applied  to  the  operator's  ability 
to  deal  with  information  which  requires  attention.  The  use  of  a  secondary 
task  of  a  non- information  nature  seeks  to  provide  an  indicator  of  how 
much  additional  work  the  operator  can  undertake.  Alternatively,  the 
secondary  task  can  induce  changes  in  the  level  of  processing  performance 
which  are  indicative  of  the  operator's  information  load. 

The  secondary  task  technique  is  a  sensitive  indicant  of  primary  task 
demand  when  perceptual  motor  activity  makes  up  the  demand  (Rolfe,  1973). 
However,  lack  of  validation  with  secondary  task  results  has  limited  this 
approach  to  essentially  one  of  a  laboratory  tool.  The  technique  has 
shortcomings  regarding  interference  with  performance  by  the  secondary 
task,  and  with  producing  confounding  factors  of  arousal  and  fatigue  as 
well  as  influencing  task  factors  such  as  information  load,  information 
speed,  rate  of  responding,  stimulus  compatibility,  and  memory  load 
(Brown,  1963). 
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A  simple  approach  is  that  of  time  line  analysis.  The  input  demand  repre¬ 
sented  by  number  of  messages  (Chu  and  Rouse,  1979)  or  by  total  MAU  values 
in  the  message  set  (Samet,  1978)  show  the  robustness  of  these  measures  in 
both  flight  management  and  political  military  scenarios.  Again,  problems 
develop  as  the  supervisory  control  situation  deals  primarily  with  top 
level,  discrete  events,  rather  than  molecular  items.  Queueing  models 
are  better  suited  to  this  domain. 

Queueing  models  have  the  ability  to  do  a  good  job  of  predicting  the  average 
fraction  of  time  the  pilot  is  busy,  (Server  occupancy  measure),  given  a 
good  description  of  event  arrival  times,  task  completion  times,  and  the 
priority  of  tasks.  Queueing  models  also  enable  the  model  prediction  of 
(momentary)  peak  load  as  a  function  of  input  demands.  In  the  most 
complicated  situations,  queueing  theory  only  provides  bounds  for  predicted 
performance  measures. 

It  was  proposed  in  Chu  and  Rouse  (1979)  that  the  operator  load  be  regulated 
through  a  set  of  thresholds  on  input  demand;  when  demand  exceeded  the 
upper  threshold,  the  upcoming  task  was  routed  to  automated  function,  while, 
when  demand  fell  below  the  lower  threshold,  the  upcoming  task  was  routed 
to  the  operator.  A  look-up  table  or  a  fast  model  had  to  be  maintained  to 
relate  the  operator  effort  and  performance  to  the  input  demand  (see 
Figure  3-6). 

Several  experimental  studies  (Chu  and  Rouse,  1979)  demonstrate  that  the 
queueing  model  is  capable  of  representing  the  multi-task  decision  making 
situation,  and  accurate  at  predicting  such  system  performance  measures 
as  delay  time  and  server  occupancy.  The  simple  measure  of  server  occupancy 
was  found  to  correlate  highly  with  the  subjective  effort  rating  in  a 
combined  monitoring  and  control  multi-task  situation.  An  adaptive  threshold 
policy  for  routing  input  demand,  which  employed  a  fast-time  model  based  on 
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a  queueing  simulation,  were  shown  to  significantly  improve  performance 
time  and  were  better  accepted  by  the  participating  subjects  compared  with 
a  fixed  threshold  policy. 

Experimental  studies  of  the  MAU  approach  to  modeling  and  dynamically 
controlling  information  flows  have  also  shown  promise.  Samet's  (1978) 
automated  information  selection  and  pacing  model  is  based  on  the  imple¬ 
mentation  of  a  secondary  task  to  dynamically  assess  the  operator's  load 
in  a  command  and  control  based  information  processing  task.  An  adaptive 
algorithm  continually  adjusted  the  pacing  rate  in  accordance  with  fluc¬ 
tuations  in  assessed  operator  load.  The  functional  block  diagram  in 
Figure  3-7  shows  the  major  components  of  the  information  selection  and 
pacing  system.  A  calibration  session  had  to  be  run  to  determine  the 
operator's  baseline  level  of  performance  on  the  secondary  task.  The 
information  load  was  estimated  not  by  the  number  of  presented  messages  but 
rather  by  the  sum  of  MAU  in  the  message  set.  The  experimental  demonstra¬ 
tion  showed  the  feasibility  of  such  a  user-based  information  selection 
and  pacing  to  maintain  information  presentation  to  the  operator  at  an 
acceptable  level. 


A  combination  of  the  MAU  and  Queueing  model  formulations  appears  to  have 
the  most  promise  in  the  advanced  aircraft  situation.  The  hybrid  approach 
would  compensate  for  operator  loading  through  the  following  three  modes: 

(1)  Adjusting  a  threshold  of  information  presentation. 

(2)  Adjusting  additive  factors  for  evaluating  the  information. 

(3)  Adjusting  the  allocation  of  responsibility  between  pilot  and 
computer. 


The  criteria  for  adjustment  would  take  the  form  of: 
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(1)  The  measured  task  performance. 

(2)  The  predicted  task  load,  from  indicators  of  complexity, 
speed,  and  task  interference. 

(3)  The  subjective  operator  input. 

Tests  were  made  in  the  experimental  study  (Section  4)  to  determine  responses 
to  the  speed  and  task  complexity  factors.  Once  the  responses  are  ascer¬ 
tained,  individualized  models  for  loading  can  be  developed. 


4.  EXPERIMENTAL  STUDY 


4.1  Overv i ew 

A  set  of  experimental  studies  were  performed  to  determine  the  efficacy  of 
alternative  configurations  of  the  adaptive  decision  model  under  different 
situational  conditions.  An  advanced  aircraft  simulation  was  employed, 
modified  slightly  from  that  used  in  the  previous  year's  studies  (Steeb, 
Davis,  Alperovitch  and  Freedy,  1978).  Individual  subjects  were  required 
to  pilot  a  simulated  aircraft  in  a  changing,  hazardous  environment.  In 
doing  so,  the  operators  were  able  to  select  from  a  variety  of  forms  of 
information  concerning  the  multiple  threats  encountered,  and  take  either 
evasive  or  aggressive  actions.  Performance  comparisons  were  made  in  the 
study  between  cost  benefit  and  multi -attribute  utility-based  models. 

Also,  the  effects  of  speed  stress  and  decision  complexity  loading  on 
model  performance  were  determined. 

4.2  Hypotheses 

The  following  experimental  hypotheses  were  tested: 

(1)  In  the  limited  resource  situation,  the  cost-benefit  model 
will  result  in  greater  overall  score,  less  cost  expenditures 
per  cycle,  and  greater  operator  acceptance  than  the  additive 
MAU  model . 

(2)  The  cost-benefit  model  will  achieve  a  greater  decision 
prediction  rate  than  the  additive  MAU  model  in  the  limited 
resource  task. 
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(3)  The  decision  models  will  exhibit  increased  variability  of 
weights  and  decreased  prediction  rates  as  the  decision 
complexity  changes  phase  by  phase  and  as  the  speed  increases. 

4.3  Task  Simulation 

The  task  simulation  is  patterned  after  an  important  and  representative 
information  acquisition  task--multiple  threat  intercept  operations  in 
advanced  aircraft.  The  simulation  is  an  adaptation  of  the  remotely 
piloted  vehicle  supervision  task  employed  in  the  previous  study  (Steeb, 
Oavis,  Alperovitch  and  Freedy,  1978).  Briefly,  the  simulation  requires 
the  operator  to  control  an  advanced  aircraft  in  a  hazardous  mission. 

Threats  of  uncertain  capability  and  location  are  encountered  repeatedly. 

The  operator  has  the  option  of  accessing  several  forms  of  information 
about  the  threats.  The  forms  of  information  differ  in  threat  discrimination 
capabilities,  transmission  costs,  processing  delays,  and  potential  of 
detection. 

4.3.1  Displays  and  Controls.  The  simulation  uses  a  computer-generated 
CRT  display,  illustrated  in  Figure  4-1.  The  environment  and  aircraft 
are  shown  as  they  would  be  in  a  moving-map  display.  Sets  of  threats 
appear  at  random  positions  at  the  upper  edge  of  the  display  and  move 
downward  at  a  constant  velocity.  The  operator  can  move  the  vehicle 
symbol  horizontally  to  one  of  eleven  different  pathways  to  avoid  the 
threats,  or  he  can  remain  on  course  and  take  an  aggressive  action  against 
one  of  the  threats.  The  actions  open  to  the  operator  are  primarily 
decision  making  in  nature.  Dynamics  of  control  are  minimized  since  the 
threat  and  vehicle  velocities  are  held  constant. 
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FIGURE  4-1  . 

SIMULATED  DISPLAY  AND  COMMUNICATIONS  PANEL 
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The  threats  introduce  both  uncertainty  and  danger  to  the  task  simulation. 
Each  type  of  threat  has  a  region  of  possible  damage  to  the  aircraft  as 
shown  in  Figure  4-2.  The  probability  of  damage  is  a  function  of  the 
horizontal  distance  between  the  threat  and  the  piloted  aircraft.  For 
ease  of  learning,  the  four  obstacle  types  are  designed  to  be  evocative 
of  the  types  of  contact  expected  to  occur  in  actual  flight  missions-- 
missiles,  fixed-wing  aircraft,  helicopters,  and  false  alarms  (birds). 

4.3.2  Information  and  Action  Options.  The  identity  and  location  of  the 
threats  can  be  determined  only  by  exercising  one  of  the  five  information 
options.  The  options  differ  in  their  ability  to  differentiate  between 
and  to  locate  the  threats.  These  capabilities  are  enumerated  in  Table 
4-1.  In  those  situations  where  the  information  option  is  unable  to 
differentiate  between  threat  types,  a  combination  symbol  is  displayed. 
Several  of  these  combination  symbols  are  shown  on  the  right  of  Table  4-1. 

After  receipt  of  the  information,  the  operator  must  take  one  of  the 
following  actions: 

(1)  Avoidance  -  route  the  aircraft  alone  one  of  the  eleven 
pathways . 

(2)  Aggression  -  continue  in  center  pathway,  and  fire  along  one 
of  the  eleven  pathways. 

Two  forms  of  outcomes  result  from  these  actions--losses  sustained  and 
tactical  gains.  Losses  result  from  damage  from  the  threats,  while 
tactical  gains  arise  from  neutralizing  the  threat.  The  amount  of  gain 
and  loss  depend  on  the  payoffs. 
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FIGURE  4-2. 


THREAT  CHARACTERISTICS: 
AIRCRAFT  (A),  MISSILE  (M) 


HELICOPTER 


(H), 
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TABLE  4-1 


INFORMATION  SOURCE  CHARACTERISTICS 


Information  Type 

Dis crimination 

Location 

1.  Full 

All 

All 

2.  Outline 

All  but  airplane/ 
helicopter 

All 

3.  Biological 

Bird  only 

All 

4.  Location 

None 

All 

Symbols 

M,  A,  H,  B 
M,  AH,  B 

MAH.  B 

X 


5.  Left/Right 


All 


Left/Right 


M,  A,  H,  B 


Sequence .  The  task  consists  of  a  series  of  similar,  connected  decisions. 
Prior  to  the  appearance  ot  any  threats,  the  operator  is  appraised  of  the 
circumstances  surrounding  the  upcoming  decision.  Me  must  then  make  an 
information  selection  by  pressing  one  of  the  buttons  shown  in  figure  4-1. 

A  set  of  two  threats  is  presented  at  the  top  of  the  screen  and  moves 
downward.  It  "full"  information  is  selected,  the  differentiated  symbols 
in  their  proper  location  move  down  the  screen.  If  "outline"  information 
is  selected,  the  missile  and  bird  symbols  are  differentiated,  but  heli¬ 
copters  and  airplances  are  represented  by  a  single,  non-differentiating 
symbol  (AM).  Similarly,  "biological"  information  will  use  a  single, 
non-di fferentiated  symbol  to  represent  either  missile,  airplane,  or 
helicopter.  "Location"  information  provides  no  discrimination.  A  symbol 
denotes  the  location  but  not  the  identify  of  threats.  "Left/Right"  infor¬ 
mation  finally,  discriminates  the  obstacles  using  the  standard  symbols, 
but  locates  them  only  as  lying  in  the  left  or  right  halt  of  the  screen. 

The  task  moves  on  continuously,  just  as  an  airborne  mission  does.  If  the 
operator  does  not  select  an  information  choice  in  the  time  allocated 
(about  5  seconds),  location  information  is  provided  by  default. 

Following  information  receipt,  the  operator  must  make  an  action  selection 
before  the  threats  reach  the  decision  limit  (a  line  approximately  C/3  of 
the  distance  down  the  screen). 

4.3.3  Situational  Conditions.  The  stages  of  an  aircraft  mission  can  be 
characterised  by  such  factors  as  danger,  difficulty,  system  reliability, 
and  communications  security.  Accordingly,  the  task  simulation  was 
designed  to  include  many  of  the  same  factors.  The  situational  conditions 
are  not  considered  to  be  experimental  variables,  but  are  factors 
contributing  to  task  complexity.  The  conditions  are: 
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(1)  Degree  of  danger  -  This  is  the  distribution  of  possible 
threats  in  a  given  phase.  A  vector  of  prior  probabilities 
of  occurrence  of  the  four  threat  types  is  assigned  to 
each  phase. 

(2)  Costs  -  a  different  cost  is  assigned  to  each  information 
choice.  This  is  the  number  of  points  expended  for  use  of 
the  information. 

(3)  Detection  -  The  increased  danger  on  the  succeeding  decision 
due  to  use  of  a  given  information  source.  An  additional 
probability  of  loss  is  associated  with  each  of  the  threats. 

(4)  Delay  -  The  delay  in  seconds  before  actual  display  of  the 
information.  This  may  also  be  quantified  as  the  percent  of 
the  distance  to  the  action  limit  before  the  information  is 
displayed. 

(5)  Payoffs  -  Different  payoffs  in  points  are  made  for  avoidance 
of  or  damage  sustained  from  the  threats,  and  for  successful 
or  unsuccessful  aggressive  actions  toward  the  threats.  Each 
of  the  payoffs  vary  phase-by-phase. 

The  presentation  of  conditions  is  organized  into  three  distinct  mission 
phases--cruise,  surveillance,  and  aggression.  Each  phase  has  set 
levels  of  danger,  detection,  and  payoffs.  For  variability,  the  costs  and 
delays  are  varied  smoothly  and  periodically  within  each  phase'.  All 
conditions  are  displayed  to  the  subjects. 
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The  phases  differ  in  decision  complexity.  Ore  phase,  aggression,  has  only 
three  non-dominated  information  selection  alternatives  (full,  outline, 
and  biological  information)  and  four  decision  attributes  (cost,  time 
delay,  avoidance  gain,  and  aggressive  gain).  The  cruise  phase  is  inter¬ 
mediate  in  complexity,  with  four  choices  and  four  attributes,  and  the 
surveillance  phase  has  maximum  complexity,  with  five  choices  and  five 
attributes.  The  phases  and  their  associated  conditions  are: 

(1)  Cruise  -  medium  payoffs  for  both  avoidance  and  aggressive 
actions;  variable  costs  and  delays;  no  possibility  of 
detection. 

(2)  Surveillance  -  high  payoffs  for  avoidance,  minimal  for 
aggressive  actions;  variable  costs  and  delays;  high 
probabilities  of  detection  with  certain  information 
sources . 


(3)  Aggression  -  high  payoffs  for  aggressive  actions  and 
delays,  minimal  for  avoidance;  variable  costs  and 
delays,  and  no  possibility  of  detection. 

Each  phase  consists  of  ten  decisions  in  sequence.  A  session  consists  of 
two  to  three  complete  sets  of  the  three  phases.  The  actual  number  of 
decisions  made  in  a  session  depends  on  the  cost  expenditures,  since 
the  task  terminates  when  the  cost  limit  is  reached. 

4.4  Decision  Model 

4.4.1  General .  The  decision  faced  by  the  operator  is  a  two-stage 
information/action  sequence,  as  shown  in  Figure  4-3.  The  decision  space 
is  fairly  large,  resulting  from  the  five  possible  information  choices. 
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22  subsequent  action  choices  (avoid  or  attack  along  each  of  the  11 
pathways),  and  330  possible  states  (combination  or  threats).  A  variety 
of  multidimensional  consequences  result  from  the  resulting  space,  stemming 
from  the  various  combinations  of  outcomes,  costs,  payoffs,  delays,  and 
future  impacts. 

A  purely  analytical  formulation  of  this  is  intractable,  just  as  it  is  for 
most  operational  information  seeking  decisions.  Categorization  is  an 
obvious  means  of  reducing  the  complexity  of  the  decision.  Here  those 
elements  in  the  decision  similar  in  consequence  can  be  classfied  together. 
For  example,  the  number  of  states  can  be  reduced  from  330  categories  to 
6  categories  by  classifying  according  to  threat  combination  (ignoring 
specific  location).  The  probability  of  each  message  and  state  then  can 
be  calculated  from  the  prior  probabilities  of  each  of  the  threats  and  the 
information  source  characteristics.  The  actions  can  be  similarly 
categorized  as  avoid  or  attack  without  regard  to  location.  Probabilities 
of  each  outcome  type--avoidance,  damage,  missed  attack,  and  hit--can 
be  established  by  observed  frequency.  To  do  so,  a  probability  estimate 
must  be  associated  with  each  combination  off  information,  message,  action, 
and  state.  After  categorization,  90  such  combinations  are  present.  These 
probabilities  were  determined  from  a  series  of  pilot  system  tests,  and  were 
intended  to  be  representative  of  the  performance  of  the  typical  subject. 
Estimates  specific  to  each  subject  were  not  made.  The  consequence  levels 
(the  attribute  level  vector)  associated  with  a  given  information  choice 
in  a  given  situation  are  calculated  by  folding  back  the  decision  tree. 

The  favored  action  choice  after  receipt  of  a  given  message  is  determined 
in  the  same  fashion. 

4.4.2  Decision  Attributes.  Five  consequence-related  attributes  are 
employed  in  the  decision  model.  The  attributes  are  the  following: 
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(Xj)  Cost  -  The  cost  of  the  communication  in  points  (costs  range 
from  0  to  13  points). 

(X2)  Delay  -  The  time  in  seconds  before  display  of  the  information 
(delays  ranged  from  0  to  5  points). 

(X3)  Detection  -  Increase  in  the  probability  of  damage  on  the 
subsequent  decision  (the  probability  increase  ranged  from 
0  to  20  percent). 

(X4)  Vehicle  loss  -  Expected  (probability  weighted)  level  of 
damage  to  own  vehicle  (range:  2  to  14  points). 

(Xjj)  Offensive  gain  -  Expected  level  of  damage  inflicted  on 
adversary  (range:  2  to  14  points). 

X^,  the  vehicle  loss  attribute  is  computed  according  to  the  following 
expression: 

X4  -  l  l  l  {P(avoid  |  y,a,z)  •  Payoff  (avoid) 

message  action  state 
y  a  z 

-P(damage)  |  y,a,z)  •  Payoff  (damage)} 


Xg,  the  offensive  gain  attribute  is  calculated  in  an  identical  manner 
using  the  probabilities  and  payoffs  for  hit  and  miss. 


4.4.3  Model  Form.  The  five  attributes  may  be  combined  into  an  evaluation 
function  using  either  of  the  two  forms  described  in  Section  3.1:  the 
linear  additive  MAU  function  or  the  cost-benefit  function.  The  MAU  function 
is  simply  the  linear  weighted  combination  of  the  five  factors: 
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MAU  C I _ ]  =  I  K. 
s  i-1  1 


(4-1) 
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Where  MAU  [I$]  is  the  aggregate  (multi-attribute)  utility  of  information 
choice  Is>  X^s  is  the  level  of  attribute  i  associated  with  information 
choice  I$,  (calculated  using  Equation  2-6)  and  K..  is  the  importance  weight 
of  attribute  i.  It  should  be  noted  that  the  program  does  not  have  access 
to  the  true  state  of  the  environment.  All  values  are  calculated  according 
to  a  priori  probability  distribution. 

The  cost-benefit  formulation  is  a  modification  of  Equation  4-1: 

i  4 

Cost-benefit  ratio  [I  ]  =  l  K.  .  X.  (4-2) 

s  s  i=l  s 

Where  C$  is  the  cost  of  information  choice  I$  and  the  other  variables  are 
defined  as  above. 

Weight  Estimation.  Similar  forms  of  adaptive  weight  estimation  were 
utilized  for  the  MAU  and  cost-benefit  models.  Adaptive  estimation  for  the 
the  MAU  models  was  performed  using  the  pattern  recognition  approach 
described  in  Section  2-3.  Prior  to  the  automated  sessions,  each  subject 
experienced  manual  selection  sessions  of  two  to  three  sequences  of  the 
three  phases  (cruise,  surveillance,  and  aggression).  The  length  of  the 
session  was  governed  by  the  information  usage.  The  session  terminated 
when  the  cost  limit  was  reached.  A  separate  5-element  weight  vector 
was  maintained  for  each  of  the  three  phases.  These  three  vectors  were 
then  "frozen"  for  use  in  the  automated  information  management  sessions 
experienced  later. 

The  estimation  process  for  the  cost-benefit  model  was  slightly  more 
involved.  When  a  discrepancy  between  predicted  and  actually  chosen 
information  sources  was  present,  the  four  attribute  weights  (see  Equation 
4-2)  were  adapted  just  as  the  five  attributes  of  the  MAU  model.  The 
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degree  of  adaptation  depended  on  the  cost  difference  between  the  pre¬ 
dicted  and  chosen  sources.  If  the  choices  were  close  in  cost,  the  same 
amount  of  movement  of  the  weights  was  prescribed  as  in  the  MAU  formulation. 
If  a  large  difference  in  cost  was  present  between  the  chosen  and  predicted 
options,  no  adjustment  was  made,  since  the  choice  may  have  simply  been 
due  to  the  cost  difference.  An  intermediate  extent  of  adjustment  was 
provided  with  moderate  differences  in  cost.  The  variable  adjustment  rule 
was  implemented  in  the  simulation  through  the  definition  of  cost  difference 
thresholds. 

4.4.4  Performance  Measures.  The  close  coupling  of  operator  and  aiding 
system  requires  evaluations  of  (1)  the  overall  system  performance  and  (2) 
the  performance  of  the  decision  model. 

System  Performance.  The  overall  system  performance  is  described  using  a 
single  index,  the  score.  The  score  is  derived  from  the  number  and  the 
cost  of  errors  committed  and  the  communications  costs  expended: 

SCORE  =  (PAYOFFS)  -  (PENALTIES  +  COMMUNICATION  COSTS) 

The  score  is  presented  to  the  subject  as  a  single  index  of  performance, 
and  his  compensation  depends  to  a  large  extent  on  the  measure.  The 
complexities  of  having  speed  as  a  second,  independent  measure  are 
avoided  by  presenting  the  task  at  a  set  pace. 

Decision  Model  Performance.  The  effectiveness  of  the  decision  model  was 
evaluated  in  terms  of  behavioral  prediction,  operator  acceptance,  and 
information  management  performance.  Prediction  refers  to  the  ability  of 
the  model  to  predict  operator  behavior  in  both  the  information  and  actual 
decisions.  Outputs  of  the  adaptive  models  were  compared  to  actual  opera¬ 
tor  choices  during  the  unaided  sessions. 
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Multivariate  analysis  of  variance  was  used  to  analyze  the  distribution  of 
information  choices  and  the  vectors  of  importance  weights  .  The 
dependent  variables  in  these  cases  are  (1)  the  frequency  with  which  each 
information  type  is  chosen,  and  (2)  the  vectors  of  by  subject  and 
phase. 

4.4.5  Subjects  and  Procedure.  The  eight  subjects  participating  in  the 
study  were  recruited  from  nearby  universities  and  military  reserve  units. 
They  represented  the  type  of  personnel  who  might  interface  with  computer- 
aided  information  systems.  The  subjects  ages  ranged  from  21  to  34,  and 
they  had  completed  between  two  and  eight  years  of  college.  Four  were 
male  and  four  female.  Five  of  the  eight  had  experience  working  with  com¬ 
puters. 

Each  subject  underwent  one  hour  of  orientation  and  practice  prior  to  the 
experimental  sessions.  The  experimental  sessions  consisted  of  four  manual 
information  selection  sessions  followed  by  four  automated  information 
selection  sessions.  The  order  of  the  sessions  is  shown  in  the  repeated 
measures  experimental  design  of  Figure  4-4.  Each  experimental  session 
consisted  of  two  to  three  complete  sequences  of  cruise,  surveillance  and 
aggression,  the  actual  number  depending  on  the  point  of  resource  depletion. 
A  session  lasted  approximately  30  minutes  and  consisted  of  60-80  decision 
cycles.  The  subjects  were  paid  $4.00  per  hour  and  given  a  bonus  to  up  to 
$4.00  per  hour  contingent  on  performance. 
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5.  EXPERIMENTAL  RESULTS  AND  DISCUSSION 

5.1  General  Observations 


The  choices  of  information  acquisition  and  action  selection  in  the  simula¬ 
tion  were  found  to  be  sufficiently  varied  and  difficult  to  provide  a  good 
test  of  model-based  information  management.  A  wide  variety  of  behaviors 
were  observed  and  modeled.  The  task  simulation  was  also  sufficiently 
demanding  to  maintain  a  high  level  of  subject  interest.  The  subjects 
learned  the  task  procedures  readily  and  by  the  end  of  the  training  session, 
could  effectively  handle  the  task  requirements  in  both  slow  and  high-speed 
conditions  and  in  manual  and  automatic  modes. 

5.2  System  Performance 

Performance  under  the  three  forms  of  information  selection--manual ,  MAU- 
based  automatic,  and  cost-benefit  automatic--were  found  to  be  essentially 
equivalent.  Significant  differences  in  behavior  were  found  between  the 
three  modes,  but  no  significant  advantages  in  overall  score,  perceived 
task  difficulty,  or  perceived  information  selection  effectiveness  were 
noted. 

A  surprising  finding  was  the  significantly  greater  cost  efficiency  deriving 
from  use  of  the  MAU  model  compared  to  that  found  with  the  cost-benefit 
model.  It  was  anticipated  that  in  the  resource-limited  task  simulation, 
the  cost-benefit  model  would  place  a  greater  emphasis  on  conserving  costs 
than  would  the  MAU  model.  In  fact,  as  shown  in  Figure  5-1,  the  MAU  model 
resulted  in  13»  more  decision  cycles  than  the  cost  benefit  model 
(F  =  9.91,  df  =  1,  4,  P  <  .05).  With  either  model,  aided  performance 
correlated  strongly  with  the  previous  manual  performance.  The  cost-benefit 
model  exhibited  a  correlation  of  .70  (P  <  .05)  between  the  manual  and 
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NUMBER  OF  DECISION  CYCLES  COMPLETED  AS  A 
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automatic  scores,  while  the  MAU  model  resulted  in  a  .61  correlation 
(P  <  .05).  In  effect,  good  manual  performance  was  necessary  for  good 
aided  performance.  The  models  did  provide  a  certain  amount  of  leveling, 
however.  The  variance  of  aided  scores  was  significantly  lower  than  the 
variance  of  the  manual  scores  (x  =  7.73,  df  =  3,  P  <  .05).  The  auto¬ 
mation  seemed  to  provide  a  smoothing  of  the  more  erratic  manual  per¬ 
formance. 

5.3  Performance  Components 

Score  performance  under  the  manual  conditions  was  found  to  be  a  function 
of  several  factors.  Significant  correlations  were  found  between  the 
score  achieved  and  (1)  the  number  of  decision  cycles  (r  =  .54,  df  =  14, 

P  <  .05),  and  (2)  the  deviation  from  expected  utility  (DEU )  with  the  MAU 
model  (r  =  .54,  df  =  14,  P  <  .05).  Essentially,  those  subjects  who  placed 
the  greatest  priority  on  cost  conservation  and  who  acted  most  consistently 
in  their  decision  policies  achieved  the  highest  scores. 

Consistency  with  respect  to  the  cost-benefit  model  did  not  appear  to  be 
as  important  to  performance.  Correlation  of  the  DEU  and  the  score  did  not 
reach  significance  with  this  model. 

5.4  Aiding  and  Task  Loading 

The  task  simulation  was  designed  to  incorporate  two  different  forms  of 
task  loading--speed  stress  and  choice  complexity.  Speed  stress  was  con¬ 
trolled  by  varying  the  time  allowed  for  the  information  and  action  choices. 
Choice  complexity  was  varied  within  each  experimental  session  by  definition 
of  the  three  phases,  each  having  different  numbers  of  attributes  and  infor¬ 
mation  choices. 
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Speed  stress  did  not  produce  the  expected  significant  differences  in 
choice  consistency  and  performance  score.  Apparently  the  two  levels  of 
speed  stress  were  not  sufficiently  different  to  evoke  major  differences 
in  behavior.  Comparisons  of  the  task  difficulty  estimates  and  the  infor¬ 
mation  effectiveness  estimates  did  show  a  significant  difference  in  per¬ 
ception  of  the  difficulty  and  choice  effectiveness  between  the  low  and 
high  speed  conditions  (Wilcoxin  matched-pairs  signed-ranks  test,  df  =  14, 

P  <  .05).  The  high-speed  stress  conditions  were  viewed  as  more  difficult 
and  the  information  obtained  was  seen  as  less  effective  than  in  the  low- 
speed  stress  conditions.  As  might  be  expected,  these  differences  were 
more  pronounced  in  the  difficult  surveillance  phase  compared  to  the  less 
complex  cruise  and  aggression  phases. 

Task  complexity,  as  represented  by  the  variation  in  difficulty  between 
the  three  phases,  resulted  in  significant  differences  in  behavior. 

Figures  5-2  and  5-3  show  the  policy  weights  and  the  distributions  of 
information  choices  for  each  of  the  three  phases:  cruise  at  the  top, 
surveillance  in  the  middle,  and  aggression  at  the  bottom.  Each  figure 
has  four  curves,  resulting  from  the  division  of  the  MAU  and  cost-benefit 
sessions  into  high  and  low-scoring  groups.  Thus  for  each  figure,  the 
four  curves  are  MAU-high  performance,  MAU-low  performance,  cost-high  per¬ 
formance  and  cost-low  performance.  It  should  be  noted  in  Figure  5-2 
that  there  is  no  cost  weight  in  the  cost-benefit  groups,  as  cost  acts  as 
a  non-weighted  dividing  factor  in  the  model. 

The  first  phase,  cruise,  is  of  relatively  low  complexity.  No  detection 
was  possible  and  the  aggression  option  was  chosen  over  the  avoidance 
option  in  most  instances.  The  straightforwardness  of  the  decisions  is 
reflected  in  both  the  policy  and  information  choice  distribution  curves. 

In  both  figures,  the  four  curves  lie  closely  together,  indicating  that 
milar  behavior  is  followed  regardless  of  the  performance  attained  or  the 
•*  1  used . 
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FIGURE  5-3. 

INFORMATION  CHOICE  DISTRIBUTION  AS  A  FUNCTION  OF 
MODEL  TYPE  AND  PERFORMANCE  FOR  a)  CRUISE, 
b)  SURVEILLANCE,  AND  c)  AGGRESSION 


The  second  phase,  surveillance,  shows  a  much  wider  variation  of  policy 
weights  and  information  choice  distributions.  The  large  variation  would 
be  expected,  since  the  surveillance  phase  is  the  most  complex  of  the 
three  phases,  and  would  be  expected  to  discriminate  between  decision 
makers  and  between  models.  Noticeable  differences  that  cut  across  all 
phases  are  found  with  the  cost  and  delay  weights  of  the  MAU  model.  These 
differences  consistently  discriminate  between  low  and  high  performance 
subjects  in  all  of  the  phases.  The  high  scoring  subjects  placed  a  greater 
emphasis  on  cost  and  less  emphasis  on  delay  compared  to  the  low-scoring 
group. 

The  final  phase,  aggression,  is  again  a  task  low  in  complexity.  No  detec¬ 
tion  was  present,  and  most  information  choices  were  of  the  first  three 
forms--full,  outline,  and  biological--as  these  afforded  the  most  informa¬ 
tion  for  aggressive  actions.  For  the  most  part,  the  policies  and  behaviors 
were  closely  bunched  except  for  one  outlying  set  of  trials,  that  of  the  low 
scorers  employing  the  MAU  model.  This  group  was  also  characterized  by 
especially  high  cost  expenditures,  and  low  consistency. 

It  was  expected  that  some  decision  makers  would  benefit  more  from  the 
MAU  model  and  some  would  be  aided  more  by  the  cost-benefit  model.  The  sub¬ 
jects  were  ranked  according  to  the  difference  in  gain  scores  achieved  with 
the  two  models.  For  each  subject,  a  gain  score  was  defined  at  automated 
score  minus  the  manual  score.  The  difference  between  the  MAU  and  cost- 
benefit  gain  scores  then  becomes  a  measure  of  the  relative  advantage  (or 
disadvantage)  of  the  MAU  model  compared  to  the  cost-benefit  model.  Divi¬ 
sion  of  the  subjects  into  MAU  favored  and  cost-benefit  favored  groups  did 
not  reveal  any  significant  behavioral  differences.  In  fact,  most  of  the 
difference  was  accountable  by  the  difference  in  manual  performance  under 
the  two  models.  The  difference  in  gain  scores  correlated  between  .60  and 
.96  (P  <  .05)  with  the  difference  in  manual  scores.  This  indicates  that 


most  of  the  model  differences  were  simply  due  to  differences  in  the 
quality  of  the  original  manual  behavior  upon  which  the  models  were 
trained. 


5.5  Information  Value 


An  advantage  of  the  MAU  model  over  the  cost-benefit  model  was  evident  in 
information  value  computation.  The  marginal  information  value  calculated 
for  each  source  (see  Section  2.4.2  for  a  description  of  this  measure)  was 
correlated  with  the  actual  score  attained  with  each  source.  The  correla¬ 
tion  was  found  to  be  higher  for  the  MAU  model  (r  =  .55,  P  <  .05)  and  than 
for  the  cost-benefit  model  (r  =  .37,  P  <  .05).  When  broken  down  by  phase, 
the  findings  are  similar,  as  shown  in  Figure  5-4.  The  correlations  are 
somewhat  higher  for  the  slow  conditions  compared  to  the  fast,  as  might  be 
expected  from  the  more  deliberate  behavior  present  in  the  slow  conditions. 

5.6  Subjective  Responses 

An  interesting  finding  was  the  difference  in  subjective  estimates  of 
informative  effectiveness  between  the  manual  and  automatic  conditions. 

In  the  manual  conditions,  a  significant  correlation  was  present  between 
the  score  attained  and  the  perceived  information  effectiveness  (r  =  .50, 
df  =  14,  P  <  .05).  In  neither  mode  of  automatic  selection,  however,  did 
the  correlation  reach  significance.  It  appears  that  the  subjects  were 
more  aware  of  information  quality  in  the  manual  information  selection 
conditions. 
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6.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  experimental  study  demonstrates  the  robustness  of  linear  models  for 
capturing  information  acquisition  behavior.  Two  very  different  models 
were  found  to  provide  essentially  equivalent  performance  in  the  advanced 
aircraft  task  simulation. 

Significant  differences  between  the  models  were  seen  in  cost  conservation. 
Surprisingly,  the  cost-benefit  model  resulted  in  more  rapid  resource 
depletion  than  the  MAU  model.  This  may  have  been  due  to  the  use  of  one 
additional  adjustable  factor  in  the  MAU  model  compared  to  the  cost-benefit 
model,  and  to  the  simpler  and  more  rapid  adjustment  algorithm  used  with 
the  MAU  model. 

Using  either  model,  automated  information  management  resulted  in  the 
classic  reduction  in  noise  or  randomness  of  the  unaided  behavior.  The 
variance  of  automated  scores  was  significantly  lower  than  that  of  the 
manual  performance  scores.  Increased  aiding  with  the  adaptive  technique 
is  also  possible.  The  probability  estimation  programs  and  attribute  weights 
were  both  "frozen"  during  the  experimental  sessions.  Dynamic  estimation 
of  outcome  probabilities  specific  to  each  subject  should  improve  the 
accuracy  of  the  attribute  level  estimates  and,  in  turn,  result  in  more 
effective  information  selection.  Similarly,  allowing  the  attribute 
weights  to  adjust  dynamically  with  the  task  demands  should  improve  per¬ 
formance. 

The  influences  of  task  load  in  the  form  of  speed  and  choice  complexity 
were  evident  in  the  form  of  altered  policies,  reduced  consistency  with 
load,  and  decreased  performance.  The  differences  were  not  as  dramatic  as 
anticipated,  and  appear  to  require  greater  ranges  of  load  to  be  of  major 
impact.  More  continuous  variation  of  loading  also  appears  necessary  to 


determine  the  functional  relationships  required  for  incorporating  load 
factors  into  the  decision  model. 

Information  value  estimates  using  the  two  models  were  found  to  be  useful. 
Consistent  with  the  other  findings,  information  value  estimates  obtained 
using  the  MAU  model  were  significantly  more  predictive  of  actual  scores 
attained  than  were  estimates  made  using  the  cost-benefit  model.  The 
estimates  were  also  more  predictive  in. the  slow  speed,  and  presumably 
more  deliberate  conditions.  A  rough  evaluation  of  a  given  information 
system  component  thus  appears  to  be  possible  by  integrating  the  calculated 
situation-specific  information  value  contributions  over  the  expected  dis¬ 
tribution  of  situations. 

A  finding  of  potential  importance  concerns  the  level  of  awareness  of  aid¬ 
ing.  Correlations  of  the  perceived  information  effectiveness  (elicited 
prior  to  each  session)  and  the  actual  score  attained  were  moderately 
strong  for  manual  information  selection  but  not  significant  for  automated 
selection.  This  may  indicate  that  in  the  active  (manual)  information 
selection  mode,  the  operator  can  easily  discriminate  good  decision  per¬ 
formance,  while  in  the  more  passive  automated  modes,  the  operator  is  not 
as  aware  of  information  quality.  Additional  research  is  needed  to  deter¬ 
mine  the  form  of  feedback  necessary  for  awareness  of  aiding  quality  in 
both  partially  and  fully  automated  systems. 
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7.  EXAMPLE  APPLICATION:  F-14  THREAT  INTERCEPT  OPERATIONS 

7.1  Background 

Many  of  the  information  management  aids  developed  in  this  program  may  be 
illustrated  in  the  context  of  F-14  threat  intercept  operations.  Percep- 
tronics  has  recently  investigated  the  possibility  of  developing  tactical 
decision  aids  to  interface  with  the  F-14  AWG-9  weapon  system.  This  work 
was  performed  under  contract  to  Naval  Air  Development  Center  (Madni,  Steeb 
and  Freedy,  1979),  and  offers  an  excellent  opportunity  to  apply  many  of 
the  information  management  techniques  derived  here.  The  weapon  system  of 
the  F-14  is  quite  sophisticated.  It  consists  of  the  AWG-9  Airborne  Wea¬ 
pon  Control  System  (AWCS),  the  LAU-934  missile  launchers  and  the  AIM-54A 
Phoenix  missile.  The  AWG-9,  in  addition  to  launching  the  Phoenix  missile, 
can  launch  other  modern  naval  air-to-air  weapons,  including  Sparrow  and 
Sidewinder  missiles  and  can  direct  firing  of  the  M-61  Vulcan  20  mm  cannon. 
The  system  has  the  capability  to  control  this  array  of  weapons  in  any 
tactical  situation,  enabling  the  F-14  to  f ul full  its  mission  of  providing 
air  superiority  and  fleet  air  defense.  The  AWG-9  utilizes  a  high-power 
pulse  doppler  radar  with  a  planar  array  antenna  which  provides  a  'look 
down'  capability  that  enables  it  to  pick  out  moving  targets  (such  as  low 
flying  missiles  and  aircraft)  from  ground  or  sea  clutter  that  normally 
obscures  targets  in  a  conventional  radar.  In  addition,  the  system  incor¬ 
porates  a  digital  computer,  controls  and  displays,  and  an  infrared  sub¬ 
system  that  provides  an  independent  search  and  track  sensor. 

7.2  Problem  Area 


An  area  of  critical  importance  is  that  of  achieving  a  launch  acquisition 
region  (LAR)  on  each  target.  A  LAR  is  a  volume  of  space  where  a  missile 
can  be  launched  with  some  expectation  of  successful  target  intercept.  A 
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LAR  is  characterized  by  an  infinite  number  of  launch  points.  It  is 
bounded  by  maximum  range,  minimum  range  and  the  doppler  exclusion  region. 
Maximum  Range  depends  on  (1)  how  far  the  missile  can  see  (target  detection 
capability)  and  (2)  how  far  the  missile  can  fly  (depends  on  the  missile's 
maximum  time  of  flight).  Minimum  Range  depends  on  (1)  minimum  missile 
guidance  time;  (2)  aircraft  antenna  gimbal  limit;  (3)  missile  antenna 
gimbal  limit;  (4)  missile-to-target  intercept  geometry  and  (5)  safety 
considerations.  Doppler  Exclusion  Region  results  from  the  fact  that 
doppler  frequency  of  target  return  close  to  that  of  main  lobe  clutter  (MLC) 
causes  regions  around  the  target  in  which  missiles  cannot  be  launched  or, 
subsequently,  guided.  When  target  tracks  appear  on  the  Target  Information 
Display  (TID)  they  are  sometimes  accompanied  with  LARs  on  those  targets 
and  sometimes  not.  A  LAR  on  a  given  target  indicates  that  that  target  is 
fair  game  from  that  point  on.  If  a  situation  arises  where  some  of  the 
targets  tracks  are  accompanied  by  LARs  while  others  are  not,  then  certain 
situation-specific  decisions  are  warranted.  The  present  level  of  aiding 
indicates  when  the  F-14  will  enter  a  LAR  or  exit  a  LAR.  It  makes  no 
attempt  to  inform  the  aircrew  why  they  don't  have  a  LAR  on  a  given  target, 
how  to  get  a  LAR  if  the  present  course  will  not  intersect  it  or  how  to 
stay  in  the  LAR  for  a  sufficient  period  of  time.  These  obvious  deficien¬ 
cies  coupled  with  upgraded  resources  in  terms  of  computer  memory  and  pro¬ 
cessing  make  the  LAR  entry  problem  a  rich  .framework  for  illustrating  the 
need  for  automated  information  analysis  and  management  programs. 

7.3  LAR  Aiding  System  Concept 

An  information  management  and  decision  aiding  system  must  be  designed 
with  a  view  to  answering  in  real-time  the  following  major  questions  for 
the  flight  officer. 
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(1)  What  am  I  up  against?  (Situation  assessment) 

(2)  Can  I  do  anything  about  it?  (Acquisition  of  information 
and  determination  of  alternative  maneuvers) 

(3)  Specifically,  what  should  I  do?  (Maneuver  selection) 

The  answers  to  these  questions  form  the  crux  of  the  aiding  system.  The 
answer  to  the  first  question  requires  a  predictive  assessment  of  the 
impending  threat.  This  assessment  includes  all  hostile  targets  with  and 
those  without  LARs.  The  answer  to  the  second  question  requires  a  diagnosis 
of  the  situation,  i.e.,  why  don't  I  have  a  LAR,  what  information  can  I 
gain  and  what  actions  should  I  take  to  acquire  one.  The  answer  to  the 
third  question  requires  an  estimation  of  the  consequences  of  each  feasi¬ 
ble  option  and  determining  the  optimum  choice  based  on  the  degree  to  which 
the  outcomes  associated  with  each  action  attain  the  desired  objectives. 

As  each  question  is  answered  the  partial  information  may  be  conveyed  to 
the  flight  officer  to  allow  interaction  with  the  system  and  also  have  a 
better  understanding  of  the  problem  enroute  to  the  final  option  recommenda¬ 
tion.  The  aiding  sequence  system  concept  is  illustrated  in  Figure  7-1. 


Close  correspondences  are  present  between  each  of  the  above  stages  and 
the  processes  described  in  Section  2.3.  The  situation  assessment  block 
in  Figure  7-1  is  equivalent  to  the  mission  situation  inputs  of  Figure  2-2. 
Here  the  importance  weights  are  defined  for  each  attribute  and  the  event 
probabilities  input.  The  alternative  generation  block  corresponds  to  the 
list  of  available  actions  phase  of  Figure  2-4.  The  system  constraint 
block  is  a  special  expansion  of  the  information  value  program.  The  con¬ 
straint  check  acts  as  a  filter,  culling  out  those  options  which  exceed  a 
system  constraint:  radar  look  angle,  range,  G-loading,  etc.  The  next 
block,  consequence  estimation,  is  the  same  as  attribute  level  estimation. 
Probability  and  outcome  level  computations  are  made  from  the  sensed  condi¬ 
tions.  The  final  function  in  Figure  7-1  is  that  of  option  evaluation. 
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FIGURE  7-1. 

AIDING  SEQUENCE  SYSTEM  CONCEPT 
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This  is  identical  to  the  MAI)  evaluation  used  in  the  information  evaluation 
program.  The  options  are  ranked  using  a  situation-specific  weighting  of 
attributes.  The  weights  are  determined  either  off-line,  through  direct 
elicitation  or  on-line,  adaptively. 

The  example  problem  selected  to  illustrate  the  application  of  the  infor¬ 
mation  management  program  is  that  of  multiple  threats  sighted  and  identi¬ 
fied  during  a  mission.  Launch  acquisition  reasons  (LARS'  are  acquired 
only  on  a  subset  of  the  targets  and  information  is  available  for  deter¬ 
mining  the  reasons  for  absence  of  a  LAR.  Finally,  sufficient  time  is 
present  for  generating  and  evaluating  maneuvers  for  acquiring  the  addi¬ 
tional  LARS.  This  time  span  is  anticipated  to  be  on  the  order  of  20  to 
60  seconds. 

7.4  Decision  Aid  Configuration 

7.4.1  Problem  Identification.  The  absence  of  a  LAR  on  an  identified 
target  may  be  due  to  (1)  the  lack  of  vehicle  capability  to  perform  the 
required  maneuver,  (2)  the  absence  of  a  LAR  intercept,  or  (3)  the  incom¬ 
patibility  of  continuous  radar  illumination  with  the  projected  fighter 
path.  Each  of  these  three  problems  may  be  identified  using  information 
available  to  the  AWG-9  program.  Tests  for  curve  fit  parameters,  geo¬ 
metric  checks  for  LAR  intercepts,  and  radar  scan  volume  tests  can  be  used 
to  pinpoint  the  problem. 

7.4.2  Problem  Display.  Display  of  the  identified  LAR  problem  may  be 
accomplished  on  the  target  information  display.  Vehicle  range  capability, 
LAR  intercept  discrepancy  and  radar  scan  volume  constraint  relative  to 
actual  target  azimuth  and  elevation  angles  may  each  be  shown  along  with 
the  predicted  aircraft  positions. 
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If  sufficient  time  is  available,  the  problem  may  also  be  displayed 
textual ly.  This  allows  display  of  exact  intercept  coordinates  and 
predicted  times  of  flight. 

7.4.3  Threat  Evaluation  Computation.  Threat  evaluation  is  done  by 

decomposing  the  threat  into  its  features,  assigning  weights  to  each 
feature  and  aggregating  the  feature  levels  times  weight  across  the  fea¬ 
tures  to  come  up  with  an  overall  evaluation  of  threat.  Features  that 
characterize  threat  are:  target  time  of  flight  to  intercept,  target 
maneuverability  and  target  weapon  capability.  Target  time  of  flight 
ranges  from  small,  medium  and  large  depending  on  discrete  ranges  of  the 
TOF :  0  <  TCF  <  e,  e  <  TOF  <  T,  and  TOF  >  T,  respectively.  Target 

maneuverability  ranges  from  high  to  medium  to  low.  Target  weapon  capa¬ 
bility  can  also  be  described  by  high,  medium  and  low.  The  weights 
assigned  to  each  of  these  features  is  keyed  to  the  tactical  situation, 
i.e.,  heavily  outnumered  (defensive  positive),  evenly  matched,  under 
matched  (offensive  posture).  The  targets  are  rank  ordered  from  maximum 
threat  to  minimum  threat  on  the  basis  of  pattern  recognition  of  the 
threat  features. 

7.4.4  Maneuver  Assessment.  The  candidate  information  acquisition  and 
system  status  change  options  must  be  tested  for  feasibility  prior  to 
evaluation.  Checks  have  to  be  made  to  determine  if  constraints  are 
exceeded  in  the  g-loading,  thrust,  turning  requirements,  time  to  maneuver, 
sensor  range  or  some  other  aerodynamic,  sensor  or  environmental  parameters 
or  if  aircraft  performance  is  stressed  to  its  limits  for  extended  dura¬ 
tions.  This  initial  check  acts  as  a  filtering  process  to  reduce  the 
number  of  options  actually  evaluated  to  those  that  are  physically  achiev¬ 
able  with  a  reasonable  change  of  success. 
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7.4.5  LAR  Entry  Decision  Criteria.  The  choice  of  information  acquisi¬ 
tion  and  subsequent  LAR  entry  maneuver  must  be  made  on  the  basis  of  key 
mission  objectives.  The  major  objectives  are  protection  of  the  carrier, 
avoiding  own  losses,  and  maximizing  tactical  gains.  Each  of  these  objec¬ 
tives  may  be  broken  down  into  measurable  attributes,  which  correspond  to 
the  degree  of  attainment  of  the  objective.  A  preliminary  set  of  attri¬ 
butes  is  defined  below: 

OBJECTIVE  ATTRIBUTES 

Minimize  threat  to  carrier  Degree  of  carrier  damage 
Maximize  tactical  gain  Fighter-equivalents  destroyed 

Minimize  own  losses  Number  of  missiles  fired 

Amount  of  fuel  expended 
Degree  of  damage  sustained 

Data  available  to  the  decision  aiding  program  are: 

(1)  Directly  sensed  information  concerning  the  environmental 
state  (weather,  terrain,  ECM,  target  track). 

(2)  The  vehicle  state  (velocity,  fuel). 

(3)  The  information  system  characteristics  (radar  mode,  target 
size  switch  setting,  scan  volume). 

(4)  Tactical  data  (characteristics  of  own  and  eneniy  aircraft 
maneuverability,  on-board  sensors  and  weapons;  information 
about  the  operations  area). 

(5)  Action  alternative  (control  responses,  weapon  deployment). 

The  importance  weights  for  each  of  the  attributes  (the  k.  in  equation  2-4) 
must  be  estimated  and  input  to  the  system  prior  to  decision  aid  operation. 
Sets  of  importance  weights  for  each  distinct  mission  phase  or  situation 
will  be  estimated  by  the  command  group.  The  estimation  may  be  made 


7-7 


adaptively,  using  a  computer  simulation  of  the  task,  or  may  entail  a 
direct  elicitation  (see  Steeb,  Davis,  Alperovitch  and  Freedy,  1978,  for 
a  description  of  direct  elicitation  techniques).  If  conflicts  occur 
between  the  estimators,  a  resolution  procedure  such  as  Delphi  or  nominal 
groups  (see  Gustafson,  et  al . ,  1973,  for  a  discussion  of  these  techniques) 
may  be  used  to  arrive  at  a  useful  consensus. 

A  simplified  decision  tree  for  the  LAR  acquisition  decision  is  shown  in 
Figure  7-2.  The  first  branching  of  the  tree  is  the  maneuver  option  open 
to  the  aircrew.  The  subsequent  uncertain  events  are  whenever  the  missing 
LAR  corresponds  to  a  high  or  low  threat  target.  Additional  events  deal 
with  whether  the  target  is  destroyed,  operational  or  the  aircrew's  own 
craft  is  lost.  The  probability  of  occurrence  of  each  of  these  events 
depend  on  the  actions  and  events  leading  up  to  the  event  along  with  the 
survival  conditions. 

With  these  considerations  in  mind,  the  decision  tree  shown  in  Figure  7-2  has 
been  developed  for  the  hypothetical  scenario  when  N  target  tracks  are  dis¬ 
played  on  the  TID,  but  only  M  of  them  have  LARS  (M  <  N).  All  threat  and 
preliminary  maneuver  assessments  have  been  performed.  The  first  branching 
shows  the  options  identified  to  be  feasible.  If  the  first  information 
acquisition  option  is  selected,  either  N  LARS,  less  than  N  LARS,  or  a  new 
threat  will  occur.  The  missing  LARS  may  in  turn  be  low  or  high  threat, 
resulting  in  the  avoidance  or  sustanining  of  a  hit  by  the  operator's  own 
craft.  If  a  hit  is  avoided,  the  flight  officer  selects  one  of  the  targets 
for  launch  and  either  scores  a  hit,  misses,  or  sustains  a  hit.  The  final 
branching  deals  with  the  damage  sustained  by  the  target.  Two  categories 
are  shown:  target  destroyed  and  tarqet  operational. 

Each  of  the  events  (denoted  by  a  circle  at  the  tree  branching  in  the  dia¬ 
gram)  has  an  associated  probability  of  occurrence.  The  probability  is  a 
function  of  the  actions  and  events  leading  to  that  point  and  to  the 
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FIGURE  7-2. 

LAR  ACQUISITION  DECISION  TREE 


situational  conditions  present.  All  of  these  data  are  accessible  to  the 
information  management  program.  Similarly,  the  value  associated  with  each 
end  point  of  the  tree  can  be  calculated  using  the  multi-attribute  form 
described  in  Section  2.3. 

With  the  utility  and  probability  information  in  hand,  it  is  a  simple 
mechanical  procedure  to  determine  which  of  the  decision  options  at  the 
root  of  the  tree  is  the  "best."  Starting  from  the  tip  nodes  and  working 
backwards,  every  internal  tree  node  can  be  assigned  a  value  based  on  the 
attributes  at  the  end  of  its  branches.  When  the  nodal  values  are  "rolled- 
back"  to  the  root  node,  the  initial  branch  with  the  highest  value  (expected 
utility)  is  recommended  for  adoption. 

It  should  be  noted  that  the  procedures  in  the  above  calculation— option 
constraint  checks,  probability  estimates,  value  aggregations,  and  tree- 
roll  back  calculations— are  all  performed  in  real-time  without  flight 
officer  interaction.  For  a  given  situation,  the  calculations  are  essentially 
"hard-wired. " 


Information  Requirements  of  the  Aiding  System 


The  probabilities  associated  with  each  of  the  events  in  Figure  7-2  are 
expected  to  be  directly  derived  from  on-board  data.  Generally,  the  F- 14 
state  vector  information  is  obtained  from  the  inertial  navigation  data 
while  the  target  state  vector  from  the  target  track  files.  The  probabili¬ 
ties  of  various  events  will  be  stored  on-board  in  the  form  of  table  lookup 
for  various  discrimininating  conditions  relating  to  the  target  and  the  F- 14 
state  vectors.  Each  of  the  event  probabilities  and  the  associated  data  for 
estimation  are  listed  below: 


P^:  Probability  that  less  than  N  LARS  after  information  acquisi¬ 
tion;  this  is  a  function  of  uncertainty  in  knowledge  of  true 
target  and  F- 14  state  vectors  and  the  complexity  of  the 
required  information  acquisition  option. 

?2‘‘  Probability  of  N  LARS  after  information  acquisition;  this  is 
a  function  of  uncertainty  in  knowledge  of  true  target  and 
F- 14  state  vectors. 

P^:  Probability  of  new  target  track  appearing  on  the  target 
information  display  without  a  LAR. 

Ps:  Probability  of  sustaining  a  hit;  this  is  a  function  of  target 
type,  target  capability,  intercept  geometry,  Vc,  own  resources 
available. 

Pa:  Probability  of  avoiding  a  hit  =  1  -  P  . 

Pht:  Pr°bability  that  target  with  no  LAR  is  a  high  threat;  this 

is  a  function  of  target  type,  target  capability  Vc,  time  to 
acquire  LAR,  decreasing  aspect  angle,  vehicle  capability,  etc. 

P^:  Probability  that  target  with  no  LAR  is  a  low  threat  =  1  -  Pht- 

P$h:  Probability  of  scoring  a  hit  on  target;  this  is  a  function  of 

fuze  type,  warhead,  vehicle  capability,  target  vulnerability, 
selected  missile  type,  intercept  geometry,  location  within 
target  LAR  (if  probabilistic  information  available  on-board), 
etc. 

Pm:  Probability  of  missing  target;  this  is  a  function  of  missile 
range  capability,  state  vector  errors,  location  within  LAR, 
etc. 
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7.6 


Feedback  Options 


Display  of  recommendations  from  the  decision  aids  described  above  come 
at  a  number  of  points  In  the  decision  aiding  circle.  Figure  7-1,  presented 
in  Section  7.3,  summarizes  the  inputs,  decision  aiding  algorithms,  and 
feedback  displays  for  the  overall  system.  The  first  display  is  that  of 
a  rank-ordering  of  targets  based  on  the  threat  they  pose.  The  next  dis¬ 
play  is  that  of  a  LAR  acquisition  problem  and  its  magnitude  for  each  of 
the  target  tracks  on  the  TID  without  LARS.  Depending  on  the  type  of  pro¬ 
blem  and  time  available,  either  graphic  or  textual  messages  are  displayed 
on  the  TID. 

The  succeeding  display  is  the  set  of  feasible  radar  and  maneuvers  options. 
This  is  a  listing  output  by  the  system  status  and  maneuver  assessment  pro¬ 
gram.  The  options  are  shown  both  in  list  form  and,  if  possible,  on  the 
target  track.  If  time  is  severely  limited,  the  display  may  be  suppressed. 

The  final  and  most  important  display  is  that  of  the  recommended  information 
acquisition  and  maneuver  options.  This  is  output  from  the  decision  evalua¬ 
tion  program.  The  recommended  radar  parameters  can  be  shown  on  the  TID. 
Also  a  ranking  of  the  remaining  options  can  be  listed. 

It  should  be  noted  that  the  display  of  aiding  information  is  driven  by 
the  situation  and  time  stress.  Under  extreme  time  stress,  only  the  threat 
and  system  status  assessments  may  be  possible.  With  great  time,  the  full 
set  of  displays  may  be  activated. 

The  sequence  of  aiding  is  closely  related  to  that  described  in  Section  2.3. 
Each  source  of  information  is  analyzed  according  to  its  potential  useful¬ 
ness  along  a  set  of  key  attributes.  The  information  usefulness  is  traced 
through  the  action  decisions.  Dolicy  weights  for  the  various  attributes 
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‘  are  determined  for  each  distinct  mission  situation  and  each  decision  maker. 

Finally,  the  system  can  provide  diagnostic  information,  make  information 
and  action  recommendations,  and  establish  a  basis  for  automated  display 
management. 
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APPENDIX  A 

SYSTEM  APPLICATION  GUIDELINES 


1.  SYSTEM  APPLICATION  GUIDELINES 


1.1  Introduction 


The  three  years  of  research  on  this  program  have  resulted  in  a  number  of 
analytic  and  experimental  findings.  This  section  is  a  compilation  of 
these  findings  arranged  by  topical  area.  The  intent  is  to  provide  a 
set  of  guidelines  to  assist  designers  of  upcoming  systems  and  to  provide 
a  framework  within  which  subsequent  findings  can  be  inserted. 

1.2  Application  Areas 

We  have  found  that  user-based  programs  for  information  evaluation  and 
management  have  primary  application  in  systems  with  (1)  complex  (multi- 
attributed)  information  choices,  (2)  costly  communications,  (3)  extensive 
machine  autonomy,  and  (4)  subjective  factors.  Among  the  types  of  systems 
sharing  these  characteristics  are  advanced  aircraft  information  systems, 
remote  manipulators,  remotely  piloted  vehicles,  and  biomedical  monitoring 
systems. 

The  area  showing  the  greatest  immediate  application  is  that  of  advanced 
tactical  aircraft.  These  aircraft  already  have  high  information  loads, 
substantial  amounts  of  automation,  and  critical  decision-making  require¬ 
ments. 

1.3  Catagories  of  Aiding 

Aiding  provided  by  the  information  value  programs  may  take  the  form  of 
problem  recognition,  situation  diagnosis,  option  selection,  and  system 
evaluation. 
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Problem  Recognition.  This  aspect  of  the  decision  process  involves  the 
monitoring  of  an  ongoing  action  on  a  set  of  descriptive  dimensions, 
comparing  the  set  against  acceptable  limits,  and  determining  that  the 
action  is  still  appropriate  to  the  situation.  Outcomes  of  these  pro¬ 
cesses  either  initiate  a  new  action,  or  modify  an  ongoing  action. 

Problem  recognition  is  a  key  function  in  airborne  tactical  operations. 
Aircraft  may  be  programmed  to  follow  a  course  unless  special  circum¬ 
stances  arise.  Response  maneuvers  or  transfer  of  control  to  the  super¬ 
visory  operator  may  be  specified  by  the  multi-attribute  decision  model. 
Again,  trade-offs  between  comnuni cation  costs,  potential  losses,  operator 
loading,  and  other  factors  are  made  by  the  model.  This  function  unburdens 
the  operator  of  repeatedly  interrogating  the  system  as  to  its  status. 

Situation  Diagnosis.  For  effective  decision  making  and  control  functions, 
the  operator  must  accurately  assess  and  integrate  the  probability  of 
occurrance  of  events.  Among  these  estimates  are  judgments  regarding 
likelihoods  of  obstacles,  detection  of  communications,  possibility  of 
visibility  loss,  etc. 

The  decision  model  performs  many  of  the  probability  aggregation  functions 
normally  required  of  the  operator— frequency  tabulations,  probability 
update  calculations,  etc.  The  program  performs  these  calculations  more 
rapidly,  reliably  and  accurately  than  is  possible  manually.  The  operator 
of  course,  may  still  provide  inputs  for  the  probability  programs  in  the 
form  of  single  item  prior  and  conditional  probabilities. 

Option  Selection.  The  options  open  to  the  decision  maker  are  information 
to  acquire  and  actions  to  select.  These  options  are  interdependent.  The 
value  of  information  is  due  to  its  impact  on  the  subsequent  action  decision. 


A-2 


Each  of  these  choices,  information  acquisition  and  action  selection,  is 
tied  to  maximization  of  overall  expected  gain  or  utility.  For  the 
operator,  simultaneous  consideration  of  multiple  alternatives  portrayed 
against  multiple  criteria  quickly  becomes  too  complex  for  easy  resolution. 
Computer  based  aggregation  of  the  various  factors  is  typically  faster 
and  more  consistent  than  is  possible  by  the  operator.  The  information 
value  model  can  be  used  directly  for  management  of  information.  The 
multi-attribute  utility  model  represents  the  policy  of  the  specific  user, 
it  has  access  to  the  factors  characterizing  each  information  choice,  and 
it  can  be  linked  to  the  onboard  information  control  system.  The  model  can 
be  configured  to  automatically  scan  the  available  information  source, 
select  the  immediately  most  useful  source,  and  display  it  to  the  operator. 

Information  Value  Analysis.  The  various  linear  models  developed  in  this 
program  provide  a  framework  for  determining  the  value  of  an  information 
system  component  or  configuration.  Two  main  types  of  evaluation  are 
possible:  additive  and  marginal.  The  additive  contribution  is  simply  the 
weighted  aggregation  of  attributes  for  a  given  information  source  summed 
across  the  expected  distribution  of  situations.  This  evaluation  form  is 
suited  for  information  sources  of  distinct  function--system  faults,  radar, 
communications,  etc. 

For  comparisons  of  information  value  within  an  information  type,  such  as 
variations  in  the  power  and  envelope  of  a  radar  scan,  the  marginal  contri¬ 
bution  is  a  useful  measure.  This  is  the  decision-by-decision  sum  of  the 
marginal  contributions  of  a  source  compared  to  the  next  most  highly  valued 
option.  The  intent  is  to  determine  if  the  source  in  question  makes  a 
justifiable  additional  benefit  beyond  the  existing  source  set.  The 
marginal  information  value  was  found  to  correlate  strongly  with  the 
actual  scores  attributable  to  each  source. 
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Implications  for  Choice  of  Aiding  Level.  The  choice  of  the  form  of 
aiding  to  be  used  in  a  given  situation  depends  on  the  time  stress  pre¬ 
sent,  the  complexity  of  the  decisions,  the  criticality  of  the  outcomes, 
and  the  amount  of  pre-analysis  possible.  As  the  time  stress  and  the 
decision  complexity  decrease,  the  usefulness  of  automated  option  selection 
decreases.  When  the  operator  has  sufficient  time  to  consider  only  a  few 
low-dimensionality  choices,  the  situation  diagnosis  form  of  aiding 
appears  most  useful,  providing  the  operator  with  data  on  the  implications 
of  the  choices.  The  first  and  second  years  of  the  study  demonstrated 
that  the  more  complex  model -based  recommendation  of  choices  or  actual 
replacement  of  the  operator  are  most  useful  in  situations  of  high  time 
stress,  decision  complexity  and  extensive  model  training  (Steeb,  et  al , 
1978,  1979).  Aiding  in  the  form  of  system  evaluation  is  most  effective 
as  a  means  of  designing  information  system  components  for  specified 
mission  requirements  and  for  adjusting  the  system  characteristics  during 
a  mission. 

1 . 4  Model  Structu re 


A  number  of  types  of  models  are  possible  for  capturing  an  operator's 
information-seeking  behavior.  Among  the  more  prominent  are  (1)  cue 
regression  models,  (2)  utility  models,  (3)  queueing  models,  and  (4)  game- 
theoretic  formulations. 

Cue  Regression.  Cue  regression  is  the  simplest  of  the  group.  In  cue 
regression,  a  single  linear  model  aggregates  a  variety  of  independent 
factors  to  predict  behavior.  Each  factor  is  scaled  and  weighted  according 
to  importance.  The  influences  of  the  individual  factors  are  then  combined 
additively,  multipl icati vely ,  or  in  some  other  fashion  to  arrive  at  an 
evaluation  of  each  information  choice.  The  formulation  for  the  simple 
additive  case  is: 
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R  •  [  HfSf  +  C 
i 

Where  R  is  an  interval  scaled  performance  response,  S..  is  the  stimulus 
level  on  dimension  i ,  Wi  is  the  regression  weight,  and  C  is  an  optional 
scaling  constant.  The  advantages  of  the  cue  regression  approach  are  its 
simplicity  and  robustness.  Virtually  any  available  predictive  factors 
can  be  used,  such  as  information  content,  data  perishability,  channel 
quality,  and  operator  load.  In  fact,  the  ensuing  action  decision  does 
not  even  have  to  be  represented.  The  terminal  decision  is  simply  assumed 
to  drive  the  operator's  information  seeking  behavior.  Also,  the  linear 
model  has  been  shown  to  be  robust  to  inaccuracies  in  modeling,  and 
highly  predictive  of  behavior  in  a  variety  of  situations  (Dawes  and 
Corrigan,  1974;  Slovik,  Fischhoff,  and  Lichtenstein,  1977).  This  model 
is  most  useful  in  situations  were  the  relationships  between  the  cues 
and  the  outcomes  of  the  action  decision  are  not  well-defined  or  where  a 
minimum  of  modeling  effort  is  justified. 

Multi -Attribute  Utility  (MAU)  Models.  The  MAU  models  make  the  information 
seeking  process  more  goal  directed,  normative  and  axiomatic.  Instead  of 
simply  attempting  to  predict  behavior  on  the  basis  of  a  set  of  independent 
features,  the  utility  models  tie  the  information  decisions  directly  to 
the  ensuing  action  decisions.  The  value  of  obtaining  information  is 
determined  by  calculating  its  impact  on  the  expected  utility  of  the  sub¬ 
sequent  action  decision.  The  information  is  assumed  to  change  the  pro¬ 
bability  distributions  of  the  consequence  sets  and,  in  turn,  to  revise 
the  expected  values  of  the  alternative  actions.  Nevertheless,  the  form 
of  the  model  is  again  a  linear  additive  rule.  The  utility  of  an  action 
is  considered  to  be  an  aggregate  of  many  possible  outcomes,  each 
expressed  along  a  set  of  attributes: 
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EU  (ak)  =  l  P(zh)  l  U.  (ak,  zh) 

states  attributes 
h  i 

Where  EU  (ak)  is  the  expected  utility  of  action  k,  P(zh)  is  the  proba¬ 
bility  of  state  occurring,  and  (ak«  z h)  is  the  utility  function  over 
the  I**1  attribute  associated  with  state  h  and  action  k.  The  formulation 
is  the  result  of  several  key  simplifying  assumptions.  The  decision  maker 
is  assumed  to  be  risk  neutral,  so  that  indifference  is  present  between 
the  expectation  across  a  set  of  uncertain  outcomes  and  the  uncertain  out¬ 
comes  themselves.  This  allows  the  probabilities  to  be  entered  as  simple 
coefficients.  Also,  the  attributes  are  assumed  to  satisfy  additive 
independence,  allowing  the  linear  additive  form  of  aggregation.  Tests 
for  compliance  with  these  assumptions  can  be  found  in  V.  Winterfeldt 
(1975)  or  Keeney  and  Raiffa  (1976). 

The  impact  of  a  message  or  item  of  data  is  to  change  the  probability 
distribution  of  the  states  z^.  Once  the  message  is  received,  a  maximum 
utility  action  a*(y)  can  be  identified.  The  expected  utility  of  selecting 
an  information  source  $  then  becomes: 

EU  (S)  =  l  l  P(zh)  P(y|zh)  u(a*(y) ,  zh) 

messages  states 
y  z 

Here  u(a*(y),  z^)  is  the  utility  of  taking  action  a*(y)  given  that  state 
z^  occurs.  The  utility  function  is  again  multi -attributed,  but  for 
simplicity  u(a*(y),  z^)  is  portrayed  as  having  already  been  aggregated 
across  the  various  dimensions. 

This  type  of  analysis  is  suited  for  highly  structured  tasks.  Not  only 
must  the  possible  states,  messages,  actions,  and  outcomes  be  specifiable. 
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but  the  prior  state  probabilities  and  the  conditional  probabilities 
characterizing  the  information  system  must  be  derivable. 

Queueing  Models.  The  cue  regression  and  multi-attribute  utility  models 
are  well  suited  to  discrete,  event-driven  decisions,  but  are  not  designed 
to  model  the  many  continuous  behaviors  present  in  monitoring,  tracking, 
etc.  Many  of  these  continuous  stochastic  processes  can  be  modeled  by 
embedding  the  multi-attribute  decision  model  in  a  queueing  model.  Here 
the  time  distributions  of  processes  such  as  system  faults,  course  errors 
and  energy  management  losses  are  known,  and  a  queue  of  potential  messages 
or  sampling  options  are  present.  The  queueing  model  provides  an  estimate 
of  the  buildup  in  uncertainty  regarding  a  given  process.  The  multi-attri¬ 
bute  decision  model  is  then  incorporated  as  a  criterion  function  in  the 
queueing  model. 

The  queueing  framework  of  the  multi -process  system  assumes  that  the  time 
distributions  of  information  message  arrival  and  service  times  are  known. 
With  the  service  discipline  specified,  the  queueing  model  would  predict 
steady-state  system  characteristics  such  as  average  delay  time,  through¬ 
put,  utilization  and  server  occupancy  (fraction  of  time  the  server  is 
occupied  by  the  message  processing).  The  queueing  model  is  also  capable 
of  modeling  multiple  display/multiple  operator  situations. 

Game  Theory  Models.  The  final  type  of  model,  the  game  theoretic  approach, 
assumes  the  presence  of  an  adversary  response  to  the  information  seeking 
decision.  Minimax  criteria  and  dynamic  programming  techniques  may  be 
used  in  this  type  of  situation,  but  quickly  become  intractable  in  situa¬ 
tions  of  any  complexity.  While  the  cue  regression,  MAU  and  queueing 
models  normally  assume  a  benign  environment,  factors  for  adversary  response 
can  be  included  in  the  criterion  functions.  For  example,  the  second  and 
third  year  studies  included  a  linear  factor  for  likelihood  of  damage  due 
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to  detection  of  ones'  own  aircraft.  This  modified  linear  model  is  much 
easier  to  deal  with  than  the  true  game- theoretic  approaches. 

1.5  Factor  Choice 

The  choice  of  attributes  making  up  the  criterion  function  of  the  cue 
regression,  MAU  or  queueing  models  has  been  found  to  be  an  extremely 
important  process.  The  attributes  represent  the  constituent  effects  of 
the  information  choices  (and  the  action  choices  in  the  MAU  and  cost- 
benefit  models).  The  effects  will  be  weighted  and  aggregated  together 
to  arrive  at  an  overall  evaluation  of  an  outcome. 

Necessary  Attribute  Characteristics.  Desirable  characteristics  of  an 
attribute  set  are  accessibility  for  measurement,  independence,  monotonicity 
with  preference,  completeness  of  the  set,  and  meaningful  ness  for  feedback. 
Monotonicity,  in  this  content,  implies  that  an  increase  in  the  attribute 
levels  are  monotonic,  a  simplication  is  possible.  Fisher  (1972)  and 
Gardiner  (1974)  note  that  a  straight  line  approximation  to  the  utility 
function  results  in  minor  losses  of  model  accuracy.  Independence  implies 
that  the  value  of  an  attribute  is  independent  of  changes  in  other  attri¬ 
butes.  Completeness  has  to  do  with  prediction:  if  the  attribute  set  is 
complete  and  the  decision  maker  consistent,  then  all  behavior  should  be 
accounted  for. 

Information  costs  may  comprise  attributes  of  special  note.  Often,  the 
benefits  of  an  Information  acquisition  are  simply  weighted  against  the 
costs  of  acquiring  the  information.  If  a  net  gain  is  anticipated, 
acquisition  of  the  information  is  considered  justified.  Often,  though, 
the  costs  themselves  are  multi-dimensional,  comprising  energy  costs,  time 
delays,  equipment  expenditures,  and  risks  of  detection.  The  scaling, 
weighting,  and  aggregating  of  these  costs  may  be  most  easily  performed 
in  combination  with  all  of  the  non-cost  attributes--tactical  gains. 
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political  impact,  etc.  Then,  trade-offs  among  each  of  the  factors  may 
be  performed  in  a  single,  consistent  operation  using  the  MAU  model.  If 
the  cost-benefit  formulation  is  used,  on  the  other  hand,  the  weighted 
aggregate  of  benefits  is  divided  by  the  weighted  aggregate  of  costs. 

As  a  rule  of  thumb,  five  attributes  appears  to  be  an  upper  limit  to  the 
number  of  factors  a  decision  maker  can  effectively  consider  (V.  Winter- 
feldt,  1975).  If  several  factors  contribute  to  one  consequence  dimension, 
these  factors  should  be  combined  using  a  single  common  scale— dollars , 
ship-equivalents,  fuel  quantity,  etc. 

Each  of  the  attributes—communications  costs,  vehicle  losses,  etc.,-- 
must  be  scaled  with  interval  properties  along  a  set  range.  The  least 
desirable  consequence  that  may  occur  is  assigned  a  level  of  zero  on  the 
scale.  The  most  desirable  consequence  is  assigned  a  level  of  one.  The 
weighting  factors  k.  should  also  be  normalized  so  that  the  overall  worst 
combination  of  factors  results  in  a  value  of  zero  and  the  overall  best 
combination  a  value  of  one. 

If  the  cost-benefit  formulation  is  used,  the  attributes  must  also  have 
radio  scale  properties,  since  ratios  are  involved  in  the  computations. 

To  ensure  ratio  properties,  the  benefit  attributes  should  be  scaled  with 
respect  to  a  zero  cost  option. 

Determination  of  the  Attribute  Set.  The  initial  selection  of  attribute 
sets  may  be  performed  by  Interview,  intuition,  or  analysis.  Protocol 
analysis  is  a  subjective  interview  technique  whereby  the  operator  intro¬ 
spective!  y  recounts  the  factors  and  procedures  which  enter  into  his 
decisions.  Consciously  considered  attributes  may  be  identified  from  this 
introspection.  A  second,  more  objective  technique  is  possible  if  the 
decision  situation  is  highly  defined.  The  feature  set  can  be  determined 


according  to  the  predictive  error  rate  and  from  correlations  of  the 
candidate  attributes  (Felson,  1975).  The  first  attribute  chosen  is  that 
with  the  lowest  expected  probability  of  error  (EPE).  The  EPE  is  the 
error  rate  that  would  result  if  the  ith  attribute  alone  were  used  as  a 
basis  for  decision  making.  The  second  feature  chosen  is  the  one  with  the 
smallest  correlation  with  the  first  attribute.  The  choice  of  the  ith 
attribute  depends  on  its  correlation  with  the  i-1  attributes  already 
chosen. 

The  attributes  in  their  raw  form  may  be  highly  non-linear.  Linear  trans¬ 
formations  to  achieve  interval  scales  are  often  warranted.  The  effect  of 
the  linear  transformation  is  normally  minor  compared  to  the  magnitudes  of 
the  test/retest  reliability  and  intersubject  differences  (Edwards  and 
Guttentag,  1975). 

1.6  Parameter  Estimation 

The  information  value  models  exhibit  a  number  of  parameters  which  must  be 
estimated:  event  probabilities,  message  characteristics,  attribute  levels, 
and  attribute  weights.  Subjective  or  objective  estimates  must  be  made  of 
each  parameter. 

Attribute  Level  Determination.  The  actual  level  of  each  of  the  attributes 
for  a  given  outcome  can  be  determined  by  mappings  between  predictive 
features  and  the  attributes.  Predictive  features  must  be  identified  which 
are  accessible  to  the  information  management  program  and  capable  of  deter¬ 
mining  the  consequence  levels.  Mappings  between  the  predictive  features 
and  the  attributes  are  either  pre-established  or  determined  by  observation 
and  adjustment. 


Attribute  Weight  Estimation.  The  policy  defining  factors  in  the  model,  the 
importance  weights  k^ ,  are  parameters  suitable  for  either  objective  or  sub¬ 
jective  estimation.  If  the  consequences  can  be  defined  along  objective 
scales  (dollars,  ship-equivalents,  etc.),  then  the  weights  could  be  derived 
by  analysis  and  input  prior  to  system  operation.  Unfortunately,  Felson 
(1975)  states  that  only  in  a  few  highly  structured  situations  can  such  an 
optimal  model  be  derived.  More  often,  the  operator's  goal  structure, 
expressed  as  importance  weights,  must  be  elicited  or  inferred  and  then 
incorporated  in  the  model. 

Off-Line  Methods.  The  operator's  subjective  weights  may  be  defined  off¬ 
line  by  elicitation  or  on-line  through  inference.  The  off-line  methods 
include  direct  elicitation  of  preference,  decomposition  of  complex  gambles 
into  hypothetical  lotteries,  and  use  of  multi-variate  methods  to  analyze 
binary  preference  expressions.  These  techniques  are  accurate  and  reliable 
in  many  circumstances,  but  they  have  a  number  of  disadvantages  when  applied 
to  operational  systems.  Typically,  these  methods  require  two  separate 
stages--asse$sment  nad  application.  Assessment  requires  an  interruption  of 
the  task  and  elicitation  of  responses  to  hypothetical  choices.  Problems 
arise  with  such  procedures  since  the  operator's  judgments  may  not  transfer 
to  the  actual  situation;  the  decision  maker  may  not  be  able  to  accurately 
verbalize  his  perference  structure;  and  the  judgments  made  in  multi-dimen¬ 
sional  choices  are  typically  responses  to  non-generalizable  extreme  values. 

The  off-line,  direct  elicitation  methods  appear  to  be  best  suited  for  those 
situations  where  repetitive  decisions  are  not  available  and  extensive 
axiomatic  tests  of  the  decisions  are  necessary. 

Unit  Weighting  Schemes.  Unit  weighting  schemes  (in  which  all  weights  ki 
are  set  equal  to  1.0)  have  been  found  to  be  quite  effective  in  certain 
circumstances.  Errors  in  the  model  form,  positive  correlations  between 


variables,  and  small  sample  sizes  all  reduce  the  predictive  capabilities 
of  differential  weights  compared  to  unit  weights  (Einhorn  and  Hogarth, 

1975).  Essentially,  the  more  precise  and  parsimonious  the  model,  the 
more  important  differential  weights  are. 

Unit  weighting  schemes  are  expected  to  see  only  minor  application  in 
aiding  advanced  aircraft  operations.  Careful  selection  of  attributes 
minimizes  intercorrelations  between  variables,  and  the  correlations  that 
do  occur  should  tend  to  be  negative.  For  example,  in  most  cases  costly 
information  is  generally  more  informative  than  inexpensive  information, 
and  equipment  attrition  tends  to  be  negatively  correlated  with  goal 
attainment.  These  circumstances  favor  inferred  weight  models.  Unit 
weighting  schemes  should  primarily  be  useful  as  starting  points  for 
estimation,  or  as  strategies  for  situations  in  which  a  great  deal  of  noise 
is  present. 

Adaptive  Weight  Estimation.  Estimation  techniques  relying  on  inference 
from  in-task  behavior  are  also  often  useful.  The  inference  techniques 
can  be  based  on  non-parametric  forms  of  pattern  recognition.  Here  a 
model  of  decision  behavior  is  assumed  and  the  parameters  of  the  model  are 
then  fitted  by  observation  and  adjustment.  Briefly,  the  technique  considers 
the  decision  maker  to  respond  to  the  characteristics  of  the  various  alter¬ 
natives  as  patterns,  classifying  them  according  to  preference.  A  linear 
discriminant  function  is  used  to  predict  the  decision  maker's  choices,  and 
when  amiss,  is  adjusted  using  error  correcting  procedures.  In  this  way, 
no  preference  ratings  or  complex  hypothetical  judgments  are  required  of  the 
operator. 

The  adaptive  nature  of  the  estimation  program  derives  from  the  error 
correcting  adjustment  mechanism.  Expected  consequence  vectors  associated 
with  each  information  source  are  input  to  the  model.  These  consequence 
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vectors  are  dotted  with  the  weight  vector,  resulting  in  evaluations  along 
a  single  utility  scale.  The  maximum  utility  choice  is  determined  and 
compared  with  the  operator's  actual  choice.  If  a  discrepancy  occurs,  the 
weight  vector  is  adjusted  according  to  the  following  rule: 


k'  *  k  +  X(x  -  x  ) 

—  —  — c  — m 

where  Jc '  is  the  updated  weight  vector,  Jc  is  the  previous  weight  vector, 

X  is  an  adjustment  constant,  x^  is  the  attribute  vector  of  the  chosen 
alternative,  and  x^  is  the  mean  attribute  vector  of  all  alternatives  ranked 
by  the  model  above  the  chosen  alternative. 

Ideally,  the  error  correction  moves  the  weight  vector  in  a  direction 
minimizing  subsequent  errors.  The  amount  of  movement  depends  on  X,  the 
adjustment  increment.  Similar  techniques  can  be  used  for  modeling  the 
action  decision. 


System  Performance  Ana lys i s 


Performance  Measures.  The  performance  of  a  complex  control  and  decision 
task  is  normally  gauged  using  a  combination  of  indices  such  as  error  rate, 
accuracy  and  speed.  For  simplicity  in  exploratory  studies,  it  is  possible 
to  use  error  rate  as  a  single  figure  of  merit  by  assigning  the  outcomes  to 
discrete  categories  and  using  a  forced-paced  presentation  of  constant 
length.  Much  of  the  work  on  which  these  guidelines  are  based  used  such 
a  format. 


Decision  Aisinq.  The  aiding  actually  provided  to  the  operator  can  be  esti¬ 
mated  in  two  ways: 


(1)  Increased  system  performance.  The  difference  in  overall 
system  performance  with  and  without  aiding  gives  a  measure 
of  the  system  improvement.  However,  this  performance  gain 
does  not  reflect  the  unburdening  provided  by  aiding. 

(2)  Unburdening.  The  unburdening  of  the  operator  may  be  seen  by 

a  change  in  secondary  task  performance  with  and  without  aiding. 
Problems  develop  with  secondary  tasks,  though,  since  two 
Independent  performance  measures  result--one  for  the  primary 
task  and  one  for  the  secondary.  The  possible  ambiguity  can 
be  lessened  by  the  cross-adaptive  technique  of  Kelley  and 
Peterson  (1969).  Here,  the  primary  task  performance  is  kept 
at  a  stable  criterion  by  varying  the  secondary  task  difficulty. 
The  secondary  task  performance  then  indicates  unburdening. 

Two  alternative  but  less  sensitive  measures  of  unburdening 
are  the  reduction  of  operator  control  time  and  decrease  in 
number  of  operator  decisions  found  with  aiding. 

Decision  Model  Performance.  The  effectiveness  of  the  decision  model  in 
inferring  decision  parameters  and  predicting  operator  behavior  can  be 
determined  by  a  number  of  methods.  Among  these  means  of  model  validation 
are  axiomatic  tests,  measures  of  prediction,  construct  validity  tests, 
and  checks  of  operator  acceptance.  Prediction  is  the  simplest  of  these. 

The  ability  of  the  adaptive  model  (and  of  any  other  model)  to  predict 
behavior  in  both  the  information  and  control  decisions  can  be  determined 
directly.  Construct  validity  tests  are  more  difficult.  These  tests  are 
made  by  comparing  the  inferred  weights  with  weights  estimated  off-line  by 
other  techniques.  The  off-line  estimation  techniques  may  involve  direct 
estimation,  paired-comparisons,  or  interpolati ve  techniques. 
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Auxiliary  Measures.  Additional  measures  of  decision  quality  and  decision 
stability  should  be  used.  The  decision  quality  measures  are  determina¬ 
tions  of  the  deviation  from  maximum  expected  utility  exhibited  by  the 
operator.  Assuming  model  accuracy,  this  is  a  measure  of  sub-optimality  of 
behavior  due  to  logical  inconsistency.  The  second  measure,  decision 
stability,  is  measured  by  the  overall  amount  of  change  of  the  estimated 
preference  structure. 

The  deviation  from  expected  utility  can  be  computed  using  the  following 
expressions. 


n 

DEU  -  £  V 
1-1 

In  effect,  this  is  a  measure  of  the  average  "distance"  between  the  pre¬ 
dicted  and  actually  chosen  alternatives. 

Subjective  Measures.  The  attitude  of  operators  toward  their  interaction 
with  the  computer  aiding  system  can  be  examined  in  structured  form  by 
means  of  rating  scales  and  in  free  form  through  voluntary  comments  and 
experimenter  observations.  Rating  scale  judgments  should  be  taken  after 
an  experimental  sequence  is  complete,  rather  than  interrupting  the  contin¬ 
uity  of  the  task  with  interim  ratings.  Scales  that  have  been  found  to  be 
useful  are: 

(1)  Competi tion/Cooperation--To  what  extent  was  the  interaction 
characterized  by  conflict? 

(2)  Aiding--How  much  unburdening  did  the  machine  provide? 

(3)  Relative  Effect1veness--What  are  your  estimates  of  the 
machine's  and  your  own  quality  of  performance? 


I 


MAU  (model  choice.)  -  MAU  (operator  choice.) 
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(4)  Automation  Predictabil ity--What  proportion  of  the  informa¬ 
tion  and  control  recommendations  were  expected? 

(5)  Satisfaction--How  satisfied  were  you  with  your  own  and  with 
the  machine's  performance? 

1 .8  Information  Feedback 

Information  Choice.  The  selection  of  the  type  of  information  for  display 
may  be  indicated  using  a  variety  of  means.  Among  these  are: 

(1)  Signal  Light.  A  discrete  light  may  be  used  to  signal  the 
recommended  choice  on  a  bank  of  switches.  This  is  well  suited 
to  situations  where  manual  overrides  are  frequent,  since  the 
control  response  is  easily  made. 

(2)  Graphic  Legend.  A  textual  display  of  the  choice  on  the  CRT 
appears  best  suited  to  fully  automated  information  management. 
The  information  sensor  characteristics  can  be  shown  along  with 
the  information  itself. 

(3)  Auditory  Signal.  An  auditory  alerting  signal  may  be  called 
for  if  Information  transfers  are  Infrequent  and  must  be 
attended  to  immediately,  e.g.,  fault  displays. 

Level  of  Confidence.  An  estimate  of  the  expected  effectiveness  of  the  in¬ 
formation  management  program  in  the  immediate  situation  is  essential. 

Two  types  of  feedback  are  possible: 

(1)  The  marginal  benefit  of  a  given  information  choice  over  the 
remaining  choices.  This  gives  some  ideas  as  to  the  strength 
of  the  choice. 
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(2)  The  confidence  associated  with  the  information  evaluation. 

This  is  dependent  on  the  experience  associated  with  the 
particular  set  of  estimating  conditions. 

Decision  Policy  Feedback.  For  training  purposes,  it  is  useful  to  display 
the  estimated  policy  weights  back  to  the  operator  while  the  task  is  being 
performed.  Such  policy  feedback  was  found  to  be  more  effective  as  a 
training  tool  than  the  simple  display  of  the  decision  outcome  (Slovick, 
Fischhoff,  and  Lichtenstein,  1977).  Display  of  the  policy  criteria  also 
puts  the  operator  more  intimately  in  contact  with  the  operation  of  the 
information  management  system.  Operator  acceptance  of  aiding  tends  to  be 
higher  when  the  operator's  values  are  seen  to  be  incorporated  in  the 
systems  decisions. 

A  second  form  of  feedback  deals  with  operator's  decision  consistencey. 

The  DEU  measure  can  be  used  to  evaluate  decision  consistency,  as  can  a 
comparison  of  strategies  between  the  information  seeking  and  subsequent 
control  decision. 

Human/Machine  Communication.  The  operator- to-machine  comprises  all  of 
the  control  actions,  parameter  inputs,  and  information  requests  made  by 
the  operator.  These  should  include: 

(1)  Direct  Manual  Control.  Manual  inputs  for  input  of  the  deci¬ 
sion  situation,  selection  of  information  sources,  and 
execution  of  control  actions  must  be  provided  in  the  same 
manner  as  in  non-automated  systems. 

(2)  Inputs  to  Automated  Information  Management  Program.  Provision 
must  be  made  for  transmitting  situational  conditions,  deci¬ 
sion  parameters,  confidence  thresholds  and  other  task  des¬ 
criptions  by  keyboard  or  other  input  device. 
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(3)  Overrides  of  Automated  Information  Selection  Decisions. 

A  simple  control  such  as  a  momentary  push-button  Is 
necessary  for  rejecting  machine  decisions. 

(4)  Information  Requests.  Queries  concerning  the  sensor  environ 
ment  or  program  state  should  be  permitted  by  keyboard  or 
other  means. 
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